[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:16 dist-test-slave-mz20 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:16 dist-test-slave-mz20 krb5kdc[395](info): set up 2 sockets
Jun 20 06:57:16 dist-test-slave-mz20 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 06:57:17.147456 388 mini_kdc.cc:121] Time spent starting KDC: real 2.282s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 06:57:17.422196 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.274s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 06:57:17.567839 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.145s user 0.003s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 06:57:17.698755 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.131s user 0.001s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/test-user.keytab.
W20260620 06:57:18.567901 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.869s user 0.002s sys 0.001s
Jun 20 06:57:18 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260620 06:57:18.729743 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.123s user 0.000s sys 0.006s
I20260620 06:57:18.731179 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34381
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:19.042865 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:19.043174 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:19.043305 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:19.050900 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:19.051030 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:19.051110 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:19.051183 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:19.051250 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:19.060988 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:19.062332 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:19.063709 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:19.076170 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:19.076236 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:19.076313 408 server_base.cc:1034] running on GCE node
W20260620 06:57:19.076131 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:19.077339 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:19.078707 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:19.079941 408 hybrid_clock.cc:648] HybridClock initialized: now 1781938639079853 us; error 74 us; skew 500 ppm
I20260620 06:57:19.080291 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:19 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:19.083206 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:19.084640 408 webserver.cc:460] Webserver started at http://127.0.97.62:44361/ using document root <none> and password file <none>
I20260620 06:57:19.085294 408 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:19.085400 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:19.085660 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:19.088905 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "99e178a21d0a4318b11ccac2173aaf00"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "4976a64ea4faf7dd5b8b5a496a1034ec"
server_key_iv: "031799e2b884814802b6ef5732e45c79"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.089577 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "99e178a21d0a4318b11ccac2173aaf00"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "4976a64ea4faf7dd5b8b5a496a1034ec"
server_key_iv: "031799e2b884814802b6ef5732e45c79"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.094419 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 06:57:19.097633 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:19.098896 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:19.099045 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "99e178a21d0a4318b11ccac2173aaf00"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "4976a64ea4faf7dd5b8b5a496a1034ec"
server_key_iv: "031799e2b884814802b6ef5732e45c79"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.099202 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:19.115559 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:19.120065 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:19.120332 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:19.138252 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34381
I20260620 06:57:19.138285 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34381 every 8 connection(s)
I20260620 06:57:19.139639 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:19.144006 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:19.145586 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 408
I20260620 06:57:19.145833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:19.147665 388 external_mini_cluster.cc:1428] Setting key 635c8c648ed0ddf771a17063403a1ec6
I20260620 06:57:19.154803 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00: Bootstrap starting.
I20260620 06:57:19.157526 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:19.158571 475 log.cc:824] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:19.161137 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00: No bootstrap required, opened a new log
I20260620 06:57:19.168520 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } }
I20260620 06:57:19.168879 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:19.168972 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e178a21d0a4318b11ccac2173aaf00, State: Initialized, Role: FOLLOWER
I20260620 06:57:19.169557 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } }
I20260620 06:57:19.169768 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:19.169881 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:19.170024 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:19.172096 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } }
I20260620 06:57:19.172549 475 leader_election.cc:302] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99e178a21d0a4318b11ccac2173aaf00; no voters:
I20260620 06:57:19.173069 475 leader_election.cc:288] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:19.173353 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 1 FOLLOWER]: Leader election won for term 1
Jun 20 06:57:19 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:19.174365 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [term 1 LEADER]: Becoming Leader. State: Replica: 99e178a21d0a4318b11ccac2173aaf00, State: Running, Role: LEADER
I20260620 06:57:19.174871 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } }
I20260620 06:57:19.175320 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:19.177796 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99e178a21d0a4318b11ccac2173aaf00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } } }
I20260620 06:57:19.178149 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:19.180629 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99e178a21d0a4318b11ccac2173aaf00. Latest consensus state: current_term: 1 leader_uuid: "99e178a21d0a4318b11ccac2173aaf00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e178a21d0a4318b11ccac2173aaf00" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34381 } } }
I20260620 06:57:19.180907 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:19.187086 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:19.188321 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:19.151349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.0.97.62:34381)
0620 06:57:19.152175 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:19.152192 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:19.153264 (+ 1072us) server_negotiation.cc:407] Connection header received
0620 06:57:19.161903 (+ 8639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:19.161922 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:19.162264 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:19.162640 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:19.165264 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.166622 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.168197 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.168888 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.169066 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:19.175364 (+ 6298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:19.175374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:19.175394 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:19.175441 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:19.179920 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:19.181047 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:19.181056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:19.181065 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:19.181291 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:19.181859 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:19.181864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:19.181868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:19.182623 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:19.185331 (+ 2708us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:19.185634 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:57:19.186120 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":374,"threads_started":1}
I20260620 06:57:19.192529 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:19.202921 485 catalog_manager.cc:1347] Generated new cluster ID: 58399794185f40bc8aa7f9f320714672
I20260620 06:57:19.203035 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:19.213421 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:19.214926 485 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:19.224725 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99e178a21d0a4318b11ccac2173aaf00: Generated new TSK 0
I20260620 06:57:19.225509 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:19.270262 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:19.604581 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:19.604940 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:19.605098 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:19.612308 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:19.612411 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:19.612567 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:19.622488 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:19.623880 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:19.625181 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:19.638007 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:19.637984 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:19.638026 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:19.639020 501 server_base.cc:1034] running on GCE node
I20260620 06:57:19.639598 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:19.640514 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:19.641703 501 hybrid_clock.cc:648] HybridClock initialized: now 1781938639641656 us; error 40 us; skew 500 ppm
I20260620 06:57:19.642006 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:19 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:19.644877 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:19.646263 501 webserver.cc:460] Webserver started at http://127.0.97.1:38131/ using document root <none> and password file <none>
I20260620 06:57:19.646957 501 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:19.647065 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:19.647357 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:19.650123 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "ff7cdb64eb6a4bfa8a36d6ee28124555"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "e0f141d124f0eed86ed1905e2dcf75bf"
server_key_iv: "6235c72e8746797fef8547e033afc075"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.650882 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ff7cdb64eb6a4bfa8a36d6ee28124555"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "e0f141d124f0eed86ed1905e2dcf75bf"
server_key_iv: "6235c72e8746797fef8547e033afc075"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.656145 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:19.659471 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:19.660782 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:57:19.660952 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "ff7cdb64eb6a4bfa8a36d6ee28124555"
format_stamp: "Formatted at 2026-06-20 06:57:19 on dist-test-slave-mz20"
server_key: "e0f141d124f0eed86ed1905e2dcf75bf"
server_key_iv: "6235c72e8746797fef8547e033afc075"
server_key_version: "encryptionkey@0"
I20260620 06:57:19.661137 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:19.681660 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:19.685500 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:19.685762 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:19.686717 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:19.688403 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:19.688476 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:19.688589 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:19.688645 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:19.718351 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40377
I20260620 06:57:19.718392 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40377 every 8 connection(s)
I20260620 06:57:19.719612 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:19.727452 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 501
I20260620 06:57:19.727640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:19.728266 388 external_mini_cluster.cc:1428] Setting key cadb6bfb0edac4f244fbba7407e55f95
Jun 20 06:57:19 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:19.739549 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:19.722035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57031 (local address 127.0.97.62:34381)
0620 06:57:19.722434 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:19.722439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:19.723350 (+ 911us) server_negotiation.cc:407] Connection header received
0620 06:57:19.725120 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:19.725125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:19.725201 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:19.725292 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:19.726804 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.727780 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.729394 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.729971 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.730093 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:19.733633 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:19.733638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:19.733641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:19.733671 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:19.736737 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:19.737404 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:19.737407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:19.737410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:19.737526 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:19.737897 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:19.737901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:19.737903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:19.738158 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:19.738283 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:19.738993 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 06:57:19.739205 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":200,"threads_started":1}
I20260620 06:57:19.740432 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:19.722505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34381 (local address 127.0.97.1:57031)
0620 06:57:19.723201 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:19.723238 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:19.724862 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:19.725534 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:19.725543 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:19.726026 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:19.726630 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:19.726655 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.727925 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.727929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:19.729266 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:19.729277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:19.730181 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:19.730188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:19.730454 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:19.731175 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:19.731212 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:19.733415 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:19.736863 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:19.736873 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:19.736890 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:19.737286 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:19.737621 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:19.737626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:19.737630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:19.737790 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:19.738272 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:19.738280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:19.738643 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 06:57:19.738871 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 06:57:19.739381 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":256,"threads_started":1}
I20260620 06:57:19.742091 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34381
I20260620 06:57:19.742377 629 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:19.743104 629 heartbeater.cc:511] Master 127.0.97.62:34381 requested a full tablet report, sending...
I20260620 06:57:19.744984 440 ts_manager.cc:194] Registered new tserver with Master: ff7cdb64eb6a4bfa8a36d6ee28124555 (127.0.97.1:40377)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 06:57:19.746387 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:19.763857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:20.088974 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:20.089250 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:20.089368 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:20.096468 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:20.096581 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:20.096738 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:20.106961 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:20.108225 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:20.109524 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:20.121654 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:20.121699 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:20.121635 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:20.122764 635 server_base.cc:1034] running on GCE node
I20260620 06:57:20.123345 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:20.124293 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:20.125519 635 hybrid_clock.cc:648] HybridClock initialized: now 1781938640125458 us; error 46 us; skew 500 ppm
I20260620 06:57:20.125801 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:20.129073 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:20.130358 635 webserver.cc:460] Webserver started at http://127.0.97.2:40845/ using document root <none> and password file <none>
I20260620 06:57:20.131035 635 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:20.131143 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:20.131414 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:20.134102 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "da63137da13541288023f52bf6fb2e84"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d355689b651e46b2eb7d589f29172922"
server_key_iv: "66361d1b173ebc862d9824706b936def"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.134827 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "da63137da13541288023f52bf6fb2e84"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d355689b651e46b2eb7d589f29172922"
server_key_iv: "66361d1b173ebc862d9824706b936def"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.139567 635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:57:20.143042 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.144351 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:57:20.144507 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "da63137da13541288023f52bf6fb2e84"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d355689b651e46b2eb7d589f29172922"
server_key_iv: "66361d1b173ebc862d9824706b936def"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.144677 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:20.160604 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:20.165091 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:20.165318 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:20.166136 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:20.167371 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:20.167443 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.167526 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:20.167574 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.199573 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37689
I20260620 06:57:20.199637 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37689 every 8 connection(s)
I20260620 06:57:20.200774 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:20.209753 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 635
I20260620 06:57:20.209921 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:20.210445 388 external_mini_cluster.cc:1428] Setting key f97f42b14f346c98c15772b5033d0308
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:20.221086 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.203214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51263 (local address 127.0.97.62:34381)
0620 06:57:20.203358 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.203362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.204489 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 06:57:20.206605 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.206611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.206698 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.206783 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.208186 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.208886 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.210904 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.211419 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.211545 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.215568 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.215573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.215576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.215613 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.218219 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.218916 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.218920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.218923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.219052 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.219446 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.219450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.219452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.219710 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.219846 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.220507 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.220726 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:57:20.221984 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.203704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34381 (local address 127.0.97.2:51263)
0620 06:57:20.204345 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:20.204381 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:20.206324 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:20.207005 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:20.207014 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:20.207509 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:20.208026 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:20.208043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.209027 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.209031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:20.210771 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:20.210785 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.211642 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.211649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:20.211922 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.212642 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:20.212670 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:20.215296 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:20.218341 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:20.218347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:20.218360 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:20.218805 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:20.219160 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:20.219163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:20.219166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:20.219334 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:20.219829 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:20.219837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:20.220172 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 06:57:20.220391 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 06:57:20.220892 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":243,"threads_started":1}
I20260620 06:57:20.223412 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34381
I20260620 06:57:20.223706 763 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:20.224380 763 heartbeater.cc:511] Master 127.0.97.62:34381 requested a full tablet report, sending...
I20260620 06:57:20.225767 440 ts_manager.cc:194] Registered new tserver with Master: da63137da13541288023f52bf6fb2e84 (127.0.97.2:37689)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:20.226501 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:20.247682 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:20.561080 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:20.561347 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:20.561441 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:20.569074 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:20.569196 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:20.569350 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:20.579238 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34381
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:20.580511 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:20.581809 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:20.594717 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:20.594769 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:20.594828 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:20.595484 768 server_base.cc:1034] running on GCE node
I20260620 06:57:20.596093 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:20.597095 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:20.598294 768 hybrid_clock.cc:648] HybridClock initialized: now 1781938640598231 us; error 53 us; skew 500 ppm
I20260620 06:57:20.598691 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:20.601656 768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:20.603596 768 webserver.cc:460] Webserver started at http://127.0.97.3:32771/ using document root <none> and password file <none>
I20260620 06:57:20.604254 768 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:20.604365 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:20.604652 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:20.607550 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "477b9792ebb449b18da39c82c09df29d"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d19f0a186e5a1e94fb5b8b3c4ae6d0a8"
server_key_iv: "b19cd57e86a0c82b1d316f372483c3df"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.608318 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "477b9792ebb449b18da39c82c09df29d"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d19f0a186e5a1e94fb5b8b3c4ae6d0a8"
server_key_iv: "b19cd57e86a0c82b1d316f372483c3df"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.613399 768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:20.616614 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.617831 768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:57:20.617972 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "477b9792ebb449b18da39c82c09df29d"
format_stamp: "Formatted at 2026-06-20 06:57:20 on dist-test-slave-mz20"
server_key: "d19f0a186e5a1e94fb5b8b3c4ae6d0a8"
server_key_iv: "b19cd57e86a0c82b1d316f372483c3df"
server_key_version: "encryptionkey@0"
I20260620 06:57:20.618137 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:20.630007 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:20.634414 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:20.634690 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:20.635535 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:20.636741 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:20.636812 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.636922 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:20.636972 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:20.668708 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41203
I20260620 06:57:20.668742 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41203 every 8 connection(s)
I20260620 06:57:20.669893 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:20.672484 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 768
I20260620 06:57:20.672642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:20.673136 388 external_mini_cluster.cc:1428] Setting key fbb52032447034bed171a11660ccfa82
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:20.689708 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.672770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60467 (local address 127.0.97.62:34381)
0620 06:57:20.672971 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.672977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.674162 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 06:57:20.676661 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.676667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.676749 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.676837 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.678608 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.679335 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.680810 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.681204 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.681309 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.684460 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.684465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.684468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.684499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.686809 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.687515 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.687519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.687522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.687642 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.688027 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.688032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.688035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.688302 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.688433 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.689152 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.689361 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:57:20.690680 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.673281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34381 (local address 127.0.97.3:60467)
0620 06:57:20.674007 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:20.674056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:20.676375 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:20.677247 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:20.677259 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:20.677865 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:20.678425 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:20.678443 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.679487 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.679493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:20.680684 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:20.680695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.681406 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.681412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:20.681643 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.682208 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:20.682228 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:20.684275 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:20.686942 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:20.686952 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:20.686969 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:20.687401 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:20.687741 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:20.687746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:20.687750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:20.687921 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:20.688439 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:20.688447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:20.688767 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 06:57:20.689008 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 06:57:20.689516 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":279,"threads_started":1}
I20260620 06:57:20.691970 896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34381
I20260620 06:57:20.692260 896 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:20.693356 896 heartbeater.cc:511] Master 127.0.97.62:34381 requested a full tablet report, sending...
I20260620 06:57:20.694694 440 ts_manager.cc:194] Registered new tserver with Master: 477b9792ebb449b18da39c82c09df29d (127.0.97.3:41203)
I20260620 06:57:20.695423 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60467
I20260620 06:57:20.701710 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:20.742362 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.727624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.0.97.1:40377)
0620 06:57:20.728678 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.728685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.728709 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:20.728761 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.728767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.728963 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.729083 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.730405 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.731472 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.732685 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.733239 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.733335 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.737245 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.737256 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.737270 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.737305 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.739803 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.740447 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.740454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.740457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.740582 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.740925 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.740929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.740931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.741246 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.741532 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.741790 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.742025 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":924,"thread_start_us":798,"threads_started":1}
W20260620 06:57:20.743561 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37214
I20260620 06:57:20.749495 629 heartbeater.cc:503] Master 127.0.97.62:34381 was elected leader, sending a full tablet report...
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:20.784209 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.770918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.97.1:40377)
0620 06:57:20.771141 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.771147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.771304 (+ 157us) server_negotiation.cc:407] Connection header received
0620 06:57:20.771465 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.771469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.771569 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.771652 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.772871 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.773556 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.774911 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.775316 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.775410 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.779327 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.779333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.779336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.779373 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.782045 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.782666 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.782670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.782672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.782780 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.783115 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.783118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.783120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.783351 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.783530 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.783764 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.783921 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 06:57:20.785569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 501
I20260620 06:57:20.797422 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.786405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.97.1:40377)
0620 06:57:20.786603 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.786608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.786924 (+ 316us) server_negotiation.cc:407] Connection header received
0620 06:57:20.787047 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.787051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.787116 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.787185 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.788232 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.788912 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.790262 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.790734 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.790825 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.792774 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.792779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.792782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.792809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.795000 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.795582 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.795586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.795588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.795702 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.796131 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.796136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.796140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.796453 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.796692 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.796955 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.797095 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:57:20.798066 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:20.877563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 501
I20260620 06:57:20.890687 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 635
Jun 20 06:57:20 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:20.908021 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:20.891479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.0.97.2:37689)
0620 06:57:20.892033 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:20.892040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:20.892065 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:57:20.892133 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:20.892139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:20.892345 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:20.892469 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:20.893496 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.894488 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.895608 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:20.896127 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:20.896216 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:20.903033 (+ 6817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:20.903049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:20.903087 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:20.903148 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:20.905542 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.906199 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.906206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.906208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.906323 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:20.906708 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:20.906711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:20.906713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:20.907017 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:20.907247 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:20.907552 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 06:57:20.907752 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
I20260620 06:57:20.909349 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:20.989216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 635
I20260620 06:57:21.007393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 768
Jun 20 06:57:21 dist-test-slave-mz20 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:21.022341 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:21.008284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.0.97.3:41203)
0620 06:57:21.008925 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:21.008932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:21.008961 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:57:21.009018 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:21.009024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:21.009233 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:21.009359 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:21.010477 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:21.011510 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:21.012757 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:21.013298 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:21.013386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:21.017267 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:21.017279 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:21.017294 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:21.017325 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:21.019821 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:21.020424 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:21.020430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:21.020433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:21.020545 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:21.020941 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:21.020945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:21.020947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:21.021260 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:21.021490 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:21.021806 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 06:57:21.022047 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":335,"threads_started":1}
I20260620 06:57:21.023567 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:21.104743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 768
I20260620 06:57:21.117638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 408
I20260620 06:57:21.118883 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:21.218400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 408
2026-06-20T06:57:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6400 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:21 dist-test-slave-mz20 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:21 dist-test-slave-mz20 krb5kdc[917](info): set up 2 sockets
Jun 20 06:57:21 dist-test-slave-mz20 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 06:57:22.210352 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:22 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:22.372443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36593
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:22.678093 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:22.678380 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:22.678478 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:22.686563 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:22.686681 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:22.686734 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:22.686792 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:22.686834 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:22.696557 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:22.697764 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:22.699057 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:22.709746 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:22.709789 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:22.709821 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:22.710861 930 server_base.cc:1034] running on GCE node
I20260620 06:57:22.711558 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:22.712859 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:22.714061 930 hybrid_clock.cc:648] HybridClock initialized: now 1781938642714012 us; error 38 us; skew 500 ppm
I20260620 06:57:22.714390 930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:22 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:22.717278 930 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:22.718659 930 webserver.cc:460] Webserver started at http://127.0.97.62:45453/ using document root <none> and password file <none>
I20260620 06:57:22.719301 930 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:22.719400 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:22.719679 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:22.722433 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "096f5ef20cc9450aba190f1d62848923"
format_stamp: "Formatted at 2026-06-20 06:57:22 on dist-test-slave-mz20"
server_key: "df24e830091217ee8952822259a42946"
server_key_iv: "36c3393c47631963b2fab2ac4aa4275d"
server_key_version: "encryptionkey@0"
I20260620 06:57:22.723353 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "096f5ef20cc9450aba190f1d62848923"
format_stamp: "Formatted at 2026-06-20 06:57:22 on dist-test-slave-mz20"
server_key: "df24e830091217ee8952822259a42946"
server_key_iv: "36c3393c47631963b2fab2ac4aa4275d"
server_key_version: "encryptionkey@0"
I20260620 06:57:22.728340 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:57:22.731644 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:22.732919 930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:57:22.733076 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "096f5ef20cc9450aba190f1d62848923"
format_stamp: "Formatted at 2026-06-20 06:57:22 on dist-test-slave-mz20"
server_key: "df24e830091217ee8952822259a42946"
server_key_iv: "36c3393c47631963b2fab2ac4aa4275d"
server_key_version: "encryptionkey@0"
I20260620 06:57:22.733265 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:22.758895 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:22.762952 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:22.763217 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:22.779062 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36593
I20260620 06:57:22.779094 996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36593 every 8 connection(s)
I20260620 06:57:22.780364 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:22.784035 997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:22.787166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 930
I20260620 06:57:22.787354 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:22.788043 388 external_mini_cluster.cc:1428] Setting key f50ec21a23383dc4a378a808738e036c
I20260620 06:57:22.796283 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923: Bootstrap starting.
I20260620 06:57:22.800132 997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:22.801463 997 log.cc:824] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:57:22 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:22.804364 997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923: No bootstrap required, opened a new log
I20260620 06:57:22.810174 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:22.789911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.0.97.62:36593)
0620 06:57:22.790696 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:22.790710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:22.790749 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:57:22.792309 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:22.792331 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:22.792755 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:22.793080 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:22.794828 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:22.796123 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:22.797788 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:22.798488 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:22.798665 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:22.803076 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:22.803084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:22.803099 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:22.803137 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:22.806054 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:22.806873 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:22.806880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:22.806889 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:22.807071 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:22.807446 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:22.807453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:22.807455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:22.807996 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:22.808203 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:22.808718 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 06:57:22.809115 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":360,"threads_started":1}
I20260620 06:57:22.812733 997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } }
I20260620 06:57:22.813073 997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:22.813158 997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096f5ef20cc9450aba190f1d62848923, State: Initialized, Role: FOLLOWER
I20260620 06:57:22.813666 997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } }
I20260620 06:57:22.813880 997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:22.813992 997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:22.814128 997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:22.816200 997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } }
I20260620 06:57:22.816649 997 leader_election.cc:302] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096f5ef20cc9450aba190f1d62848923; no voters:
I20260620 06:57:22.817220 997 leader_election.cc:288] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:22.817329 1002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:22.817900 1002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [term 1 LEADER]: Becoming Leader. State: Replica: 096f5ef20cc9450aba190f1d62848923, State: Running, Role: LEADER
I20260620 06:57:22.818387 1002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } }
I20260620 06:57:22.819094 997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:22.820902 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096f5ef20cc9450aba190f1d62848923. Latest consensus state: current_term: 1 leader_uuid: "096f5ef20cc9450aba190f1d62848923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } } }
I20260620 06:57:22.821242 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:22.822176 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096f5ef20cc9450aba190f1d62848923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096f5ef20cc9450aba190f1d62848923" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36593 } } }
I20260620 06:57:22.822602 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:22.824741 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:22.829762 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:22.838337 1008 catalog_manager.cc:1347] Generated new cluster ID: 7d302590f6ff4903bea3b9ebe0e7d7f5
I20260620 06:57:22.838454 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:22.856992 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:22.858281 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:22.872790 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096f5ef20cc9450aba190f1d62848923: Generated new TSK 0
I20260620 06:57:22.873656 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:22.914212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:23.219300 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:23.219599 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:23.219741 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:23.227002 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:23.227114 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:23.227267 1023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:23.236729 1023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:23.238014 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:23.239331 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:23.251291 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:23.251353 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:23.251379 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:23.252136 1023 server_base.cc:1034] running on GCE node
I20260620 06:57:23.252681 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:23.253572 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:23.254770 1023 hybrid_clock.cc:648] HybridClock initialized: now 1781938643254712 us; error 47 us; skew 500 ppm
I20260620 06:57:23.255080 1023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:23 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:23.257905 1023 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:23.259258 1023 webserver.cc:460] Webserver started at http://127.0.97.1:33619/ using document root <none> and password file <none>
I20260620 06:57:23.259891 1023 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:23.260000 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:23.260279 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:23.263058 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "91e918da5b99b62155cb0f913bcbfdeb"
server_key_iv: "1a347458802cb1b9cd7f3394f7cc73a1"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.263779 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "91e918da5b99b62155cb0f913bcbfdeb"
server_key_iv: "1a347458802cb1b9cd7f3394f7cc73a1"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.268666 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 06:57:23.272082 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.273391 1023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:57:23.273554 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "91e918da5b99b62155cb0f913bcbfdeb"
server_key_iv: "1a347458802cb1b9cd7f3394f7cc73a1"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.273813 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:23.291283 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:23.295609 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:23.295881 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:23.296833 1023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:23.298111 1023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:23.298187 1023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.298305 1023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:23.298355 1023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.328895 1023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36733
I20260620 06:57:23.328943 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36733 every 8 connection(s)
I20260620 06:57:23.330137 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:23.340566 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1023
I20260620 06:57:23.340759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:23.341488 388 external_mini_cluster.cc:1428] Setting key bbc332f071b39c0b7fe125bb11e1d7c1
Jun 20 06:57:23 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:23.349830 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:23.332668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36615 (local address 127.0.97.62:36593)
0620 06:57:23.333068 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:23.333073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:23.333974 (+ 901us) server_negotiation.cc:407] Connection header received
0620 06:57:23.335784 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:23.335789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:23.335866 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:23.335956 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:23.337563 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.338314 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.339831 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.340247 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.340351 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:23.344036 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:23.344042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:23.344047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:23.344083 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:23.347085 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:23.347730 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:23.347745 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:23.347749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:23.347880 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:23.348323 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:23.348326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:23.348328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:23.348575 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:23.348678 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:23.349428 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 06:57:23.349577 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":206,"threads_started":1}
I20260620 06:57:23.351233 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:23.333139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36593 (local address 127.0.97.1:36615)
0620 06:57:23.333813 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:23.333851 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:23.335518 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:23.336204 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:23.336212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:23.336745 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:23.337369 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:23.337387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.338442 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.338445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:23.339682 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:23.339694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.340473 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.340480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:23.340766 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:23.341453 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:23.341490 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:23.343827 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:23.347205 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:23.347211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:23.347224 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:23.347619 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:23.347997 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:23.348002 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:23.348006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:23.348228 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:23.348682 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:23.348689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:23.349099 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 06:57:23.349369 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 06:57:23.349884 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":268,"threads_started":1}
I20260620 06:57:23.352723 1151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36593
I20260620 06:57:23.352998 1151 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:23.353659 1151 heartbeater.cc:511] Master 127.0.97.62:36593 requested a full tablet report, sending...
I20260620 06:57:23.356040 962 ts_manager.cc:194] Registered new tserver with Master: 0745651fee8e4a3e82fa2e18a5b3a3ea (127.0.97.1:36733)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:23.358068 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:23.378981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:23.685290 1157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:23.685664 1157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:23.685837 1157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:23.693265 1157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:23.693403 1157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:23.693560 1157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:23.703249 1157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:23.704509 1157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:23.705797 1157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:23.717947 1162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:23.717903 1165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:23.718005 1163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:23.724116 1157 server_base.cc:1034] running on GCE node
I20260620 06:57:23.724700 1157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:23.725638 1157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:23.726873 1157 hybrid_clock.cc:648] HybridClock initialized: now 1781938643726828 us; error 32 us; skew 500 ppm
I20260620 06:57:23.727221 1157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:23 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:23.730278 1157 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:23.731813 1157 webserver.cc:460] Webserver started at http://127.0.97.2:41253/ using document root <none> and password file <none>
I20260620 06:57:23.732513 1157 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:23.732628 1157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:23.732928 1157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:23.735823 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "73936a5b2a844898bb5491fa9721baa0"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "9786658e91f3bd6bd8efd7fc9ef1d2ec"
server_key_iv: "968536992230fca7ca37324894b14104"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.736651 1157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "73936a5b2a844898bb5491fa9721baa0"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "9786658e91f3bd6bd8efd7fc9ef1d2ec"
server_key_iv: "968536992230fca7ca37324894b14104"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.741720 1157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 06:57:23.745117 1172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.746503 1157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:23.746678 1157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "73936a5b2a844898bb5491fa9721baa0"
format_stamp: "Formatted at 2026-06-20 06:57:23 on dist-test-slave-mz20"
server_key: "9786658e91f3bd6bd8efd7fc9ef1d2ec"
server_key_iv: "968536992230fca7ca37324894b14104"
server_key_version: "encryptionkey@0"
I20260620 06:57:23.746897 1157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:23.763278 1157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:23.766945 1157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:23.767161 1157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:23.767989 1157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:23.769213 1157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:23.769282 1157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.769368 1157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:23.769408 1157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:23.805154 1157 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38811
I20260620 06:57:23.805199 1284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38811 every 8 connection(s)
I20260620 06:57:23.806340 1157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:23.813469 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1157
I20260620 06:57:23.813640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:23.814116 388 external_mini_cluster.cc:1428] Setting key bdac4fa4bbd99741f2c5fdd6b4dbf8c6
Jun 20 06:57:23 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:23.826102 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:23.808826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59715 (local address 127.0.97.62:36593)
0620 06:57:23.809009 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:23.809014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:23.810199 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 06:57:23.812009 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:23.812014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:23.812092 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:23.812174 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:23.813913 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.814793 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.816660 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.817042 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.817135 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:23.820899 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:23.820904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:23.820907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:23.820937 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:23.823278 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:23.823925 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:23.823930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:23.823934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:23.824059 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:23.824449 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:23.824454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:23.824457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:23.824702 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:23.824823 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:23.825571 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 06:57:23.825785 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:57:23.827095 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:23.809308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36593 (local address 127.0.97.2:59715)
0620 06:57:23.810037 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:23.810080 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:23.811742 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:23.812445 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:23.812454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:23.812976 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:23.813740 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:23.813765 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.814941 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.814946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:23.816527 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:23.816543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:23.817253 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:23.817259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:23.817540 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:23.818197 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:23.818224 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:23.820697 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:23.823407 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:23.823416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:23.823433 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:23.823816 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:23.824153 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:23.824158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:23.824162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:23.824350 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:23.824814 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:23.824821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:23.825248 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 06:57:23.825500 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 06:57:23.825997 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260620 06:57:23.828601 1285 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36593
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:23.828954 1285 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:23.829633 1285 heartbeater.cc:511] Master 127.0.97.62:36593 requested a full tablet report, sending...
I20260620 06:57:23.830999 962 ts_manager.cc:194] Registered new tserver with Master: 73936a5b2a844898bb5491fa9721baa0 (127.0.97.2:38811)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 06:57:23.831859 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:23.850198 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:24.158983 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:24.159252 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:24.159373 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:24.166414 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:24.166517 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:24.166692 1290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:24.176092 1290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:24.177356 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:24.178814 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:24.191402 1295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:24.191431 1296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:24.191402 1298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:24.192447 1290 server_base.cc:1034] running on GCE node
I20260620 06:57:24.193091 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:24.194082 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:24.195274 1290 hybrid_clock.cc:648] HybridClock initialized: now 1781938644195226 us; error 44 us; skew 500 ppm
I20260620 06:57:24.195580 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:24.198504 1290 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:24.199877 1290 webserver.cc:460] Webserver started at http://127.0.97.3:40853/ using document root <none> and password file <none>
I20260620 06:57:24.200521 1290 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:24.200632 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:24.200909 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:24.203697 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "3824ca1d95ae486aa8b8a22e465f4588"
format_stamp: "Formatted at 2026-06-20 06:57:24 on dist-test-slave-mz20"
server_key: "e66aa466f405727267579b730786cc70"
server_key_iv: "338dfa26eada88b4ac9e6ade9cee6b3c"
server_key_version: "encryptionkey@0"
I20260620 06:57:24.204427 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3824ca1d95ae486aa8b8a22e465f4588"
format_stamp: "Formatted at 2026-06-20 06:57:24 on dist-test-slave-mz20"
server_key: "e66aa466f405727267579b730786cc70"
server_key_iv: "338dfa26eada88b4ac9e6ade9cee6b3c"
server_key_version: "encryptionkey@0"
I20260620 06:57:24.209255 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:24.212599 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:24.214157 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:24.214313 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "3824ca1d95ae486aa8b8a22e465f4588"
format_stamp: "Formatted at 2026-06-20 06:57:24 on dist-test-slave-mz20"
server_key: "e66aa466f405727267579b730786cc70"
server_key_iv: "338dfa26eada88b4ac9e6ade9cee6b3c"
server_key_version: "encryptionkey@0"
I20260620 06:57:24.214524 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:24.256906 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:24.260797 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:24.261034 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:24.261896 1290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:24.263178 1290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:24.263247 1290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:24.263325 1290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:24.263365 1290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:24.292639 1290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33765
I20260620 06:57:24.292690 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33765 every 8 connection(s)
I20260620 06:57:24.294006 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:24.295249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1290
I20260620 06:57:24.295413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:24.295915 388 external_mini_cluster.cc:1428] Setting key cc408e4cde2f58584d7db1592dace65a
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:24.313992 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.296863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33383 (local address 127.0.97.62:36593)
0620 06:57:24.297063 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.297068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.298448 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 06:57:24.300281 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.300285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.300351 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.300451 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.301871 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.302641 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.304217 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.304619 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.304713 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.308145 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.308151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.308154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.308187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.310999 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.311731 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.311737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.311740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.311857 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.312262 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.312267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.312270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.312538 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.312664 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.313379 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.313627 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:57:24.315071 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.297507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36593 (local address 127.0.97.3:33383)
0620 06:57:24.298282 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.298338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.300032 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.300657 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.300665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.301159 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:24.301710 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.301727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.302799 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.302803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.304077 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.304092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.304791 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.304795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.305006 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.305661 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:24.305684 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:24.307940 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:24.311133 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:24.311143 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:24.311163 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:24.311616 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:24.311961 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:24.311966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:24.311970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:24.312159 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:24.312665 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:24.312675 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:24.313019 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.313272 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.313836 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":321,"threads_started":1}
I20260620 06:57:24.316416 1418 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36593
I20260620 06:57:24.316715 1418 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:24.317404 1418 heartbeater.cc:511] Master 127.0.97.62:36593 requested a full tablet report, sending...
I20260620 06:57:24.318913 962 ts_manager.cc:194] Registered new tserver with Master: 3824ca1d95ae486aa8b8a22e465f4588 (127.0.97.3:33765)
I20260620 06:57:24.319690 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33383
I20260620 06:57:24.323645 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:57:24.341977 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.329615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.0.97.62:36593)
0620 06:57:24.329779 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.329796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.330279 (+ 483us) server_negotiation.cc:407] Connection header received
0620 06:57:24.330448 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.330454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.330521 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.330629 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.332490 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.333175 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.334381 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.334840 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.334944 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.337162 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.337168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.337171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.337207 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.339596 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.340174 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.340180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.340184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.340303 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.340641 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.340646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.340650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.340953 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.341156 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.341506 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.341696 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:57:24.347656 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:24.354528 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:24.361102 1151 heartbeater.cc:503] Master 127.0.97.62:36593 was elected leader, sending a full tablet report...
I20260620 06:57:24.375535 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.365703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.0.97.2:38811)
0620 06:57:24.367782 (+ 2079us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.367791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.367816 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:57:24.367926 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.367933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.368185 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.369144 (+ 959us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:24.369342 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.370799 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.372895 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.374269 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.374425 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.374640 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.375045 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.375212 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1932,"thread_start_us":2058,"threads_started":1}
I20260620 06:57:24.375560 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.365594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38811 (local address 127.0.0.1:58076)
0620 06:57:24.366334 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.366385 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.366586 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.368406 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.368413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.368456 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:24.368935 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.368947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.370968 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.370973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.372749 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.372763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.374510 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.374516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.374798 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.374848 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.374978 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.375147 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":25,"thread_start_us":320,"threads_started":1}
I20260620 06:57:24.376924 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.366494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36364 (local address 127.0.97.1:36733)
0620 06:57:24.369321 (+ 2827us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.369328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.369351 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:57:24.369430 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.369437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.369659 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.370040 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:24.371158 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.372468 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.374734 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.375957 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.376115 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.376314 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.376501 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.376644 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2681,"thread_start_us":2556,"threads_started":1}
I20260620 06:57:24.377389 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.365967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36733 (local address 127.0.0.1:36364)
0620 06:57:24.367247 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.367268 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.367369 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.370514 (+ 3145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.370518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.370563 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:24.370969 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.370980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.372576 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.372581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.374605 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.374621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.376161 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.376166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.376318 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.376350 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.376431 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.376548 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"mutex_wait_us":40,"thread_start_us":1045,"threads_started":1}
I20260620 06:57:24.378906 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.366741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.0.1:40770)
0620 06:57:24.370616 (+ 3875us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.370635 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.370815 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.371617 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.371621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.371651 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:24.372008 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.372018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.373572 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.373577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.375947 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.375958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.378171 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.378176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.378334 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.378364 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.378436 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.378575 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3722,"mutex_wait_us":2,"thread_start_us":3601,"threads_started":1}
I20260620 06:57:24.379206 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.367250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.97.3:33765)
0620 06:57:24.368369 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.368377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.370719 (+ 2342us) server_negotiation.cc:407] Connection header received
0620 06:57:24.371004 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.371009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.371195 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.371341 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:24.372188 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.373413 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.376081 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.377684 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.377793 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.378647 (+ 854us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.378724 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.378877 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":964,"thread_start_us":804,"threads_started":1}
I20260620 06:57:24.379910 1220 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4731edb25547bba06c6de5039036a3 (DEFAULT_TABLE table=test-table [id=e6a2c89581c5448d922829e3ca8792d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:24.381376 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4731edb25547bba06c6de5039036a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:24.381143 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4731edb25547bba06c6de5039036a3 (DEFAULT_TABLE table=test-table [id=e6a2c89581c5448d922829e3ca8792d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:24.381901 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4731edb25547bba06c6de5039036a3 (DEFAULT_TABLE table=test-table [id=e6a2c89581c5448d922829e3ca8792d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:24.382498 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4731edb25547bba06c6de5039036a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:24.383131 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4731edb25547bba06c6de5039036a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:24.393930 1436 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Bootstrap starting.
I20260620 06:57:24.395823 1437 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Bootstrap starting.
I20260620 06:57:24.399004 1437 tablet_bootstrap.cc:654] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:24.399003 1436 tablet_bootstrap.cc:654] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:24.399202 1438 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Bootstrap starting.
I20260620 06:57:24.400259 1436 log.cc:824] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:24.400259 1437 log.cc:824] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:24.402735 1438 tablet_bootstrap.cc:654] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:24.403163 1437 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: No bootstrap required, opened a new log
I20260620 06:57:24.403427 1437 ts_tablet_manager.cc:1397] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 06:57:24.403648 1438 log.cc:824] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:24.403965 1436 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: No bootstrap required, opened a new log
I20260620 06:57:24.404207 1436 ts_tablet_manager.cc:1397] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 06:57:24.406404 1438 tablet_bootstrap.cc:492] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: No bootstrap required, opened a new log
I20260620 06:57:24.406651 1438 ts_tablet_manager.cc:1397] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 06:57:24.413467 1438 raft_consensus.cc:348] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.413822 1438 raft_consensus.cc:374] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:24.413522 1437 raft_consensus.cc:348] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.413929 1438 raft_consensus.cc:729] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0745651fee8e4a3e82fa2e18a5b3a3ea, State: Initialized, Role: FOLLOWER
I20260620 06:57:24.413978 1437 raft_consensus.cc:374] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:24.414173 1437 raft_consensus.cc:729] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73936a5b2a844898bb5491fa9721baa0, State: Initialized, Role: FOLLOWER
I20260620 06:57:24.414577 1438 consensus_queue.cc:260] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.414824 1437 consensus_queue.cc:260] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.414819 1436 raft_consensus.cc:348] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.415234 1436 raft_consensus.cc:374] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:24.415372 1436 raft_consensus.cc:729] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3824ca1d95ae486aa8b8a22e465f4588, State: Initialized, Role: FOLLOWER
I20260620 06:57:24.416119 1436 consensus_queue.cc:260] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.417415 1438 ts_tablet_manager.cc:1428] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260620 06:57:24.418390 1418 heartbeater.cc:503] Master 127.0.97.62:36593 was elected leader, sending a full tablet report...
I20260620 06:57:24.419246 1436 ts_tablet_manager.cc:1428] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 06:57:24.420316 1285 heartbeater.cc:503] Master 127.0.97.62:36593 was elected leader, sending a full tablet report...
I20260620 06:57:24.420894 1437 ts_tablet_manager.cc:1428] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260620 06:57:24.432749 1442 raft_consensus.cc:482] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:24.433013 1442 raft_consensus.cc:504] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.434473 1442 leader_election.cc:288] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73936a5b2a844898bb5491fa9721baa0 (127.0.97.2:38811), 3824ca1d95ae486aa8b8a22e465f4588 (127.0.97.3:33765)
I20260620 06:57:24.441087 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.434784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52153 (local address 127.0.97.2:38811)
0620 06:57:24.434946 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.434952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.435422 (+ 470us) server_negotiation.cc:407] Connection header received
0620 06:57:24.435647 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.435652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.435732 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.435835 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:24.436433 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.437615 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.439538 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.440256 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.440369 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.440457 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.440734 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.440847 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:57:24.441115 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.435604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52315 (local address 127.0.97.3:33765)
0620 06:57:24.435796 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.435802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.436144 (+ 342us) server_negotiation.cc:407] Connection header received
0620 06:57:24.436285 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.436289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.436342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.436417 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:24.436891 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.437606 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.439540 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.440277 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.440384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.440479 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.440781 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.440898 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:57:24.441234 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.435497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33765 (local address 127.0.97.1:52315)
0620 06:57:24.436052 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.436073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.436185 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.436448 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.436452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.436477 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:24.436759 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.436767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.437733 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.437739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.439401 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.439413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.440521 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.440527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.440649 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.440671 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.440746 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.440869 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":69,"thread_start_us":236,"threads_started":1}
I20260620 06:57:24.441411 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.434809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38811 (local address 127.0.97.1:52153)
0620 06:57:24.435331 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:24.435345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:24.435474 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:24.435881 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:24.435890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:24.435916 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:24.436311 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.436322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.437765 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.437770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.439395 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:24.439406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.440475 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.440482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:24.440604 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.440626 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:57:24.440702 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:57:24.440821 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":28,"thread_start_us":304,"threads_started":1}
I20260620 06:57:24.442101 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4731edb25547bba06c6de5039036a3" candidate_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3824ca1d95ae486aa8b8a22e465f4588" is_pre_election: true
I20260620 06:57:24.442131 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4731edb25547bba06c6de5039036a3" candidate_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73936a5b2a844898bb5491fa9721baa0" is_pre_election: true
I20260620 06:57:24.442525 1240 raft_consensus.cc:2454] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0745651fee8e4a3e82fa2e18a5b3a3ea in term 0.
I20260620 06:57:24.442533 1373 raft_consensus.cc:2454] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0745651fee8e4a3e82fa2e18a5b3a3ea in term 0.
I20260620 06:57:24.443096 1041 leader_election.cc:302] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745651fee8e4a3e82fa2e18a5b3a3ea, 3824ca1d95ae486aa8b8a22e465f4588; no voters:
I20260620 06:57:24.443413 1442 raft_consensus.cc:2781] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:24.443524 1442 raft_consensus.cc:482] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:24.443619 1442 raft_consensus.cc:3037] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:24.445919 1442 raft_consensus.cc:504] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.446419 1442 leader_election.cc:288] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [CANDIDATE]: Term 1 election: Requested vote from peers 73936a5b2a844898bb5491fa9721baa0 (127.0.97.2:38811), 3824ca1d95ae486aa8b8a22e465f4588 (127.0.97.3:33765)
I20260620 06:57:24.446794 1240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4731edb25547bba06c6de5039036a3" candidate_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73936a5b2a844898bb5491fa9721baa0"
I20260620 06:57:24.446862 1373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4731edb25547bba06c6de5039036a3" candidate_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3824ca1d95ae486aa8b8a22e465f4588"
I20260620 06:57:24.446946 1240 raft_consensus.cc:3037] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:24.447039 1373 raft_consensus.cc:3037] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:24.449343 1373 raft_consensus.cc:2454] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0745651fee8e4a3e82fa2e18a5b3a3ea in term 1.
I20260620 06:57:24.449455 1240 raft_consensus.cc:2454] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0745651fee8e4a3e82fa2e18a5b3a3ea in term 1.
I20260620 06:57:24.449726 1041 leader_election.cc:302] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0745651fee8e4a3e82fa2e18a5b3a3ea, 3824ca1d95ae486aa8b8a22e465f4588; no voters:
I20260620 06:57:24.450006 1442 raft_consensus.cc:2781] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:24.450394 1442 raft_consensus.cc:686] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [term 1 LEADER]: Becoming Leader. State: Replica: 0745651fee8e4a3e82fa2e18a5b3a3ea, State: Running, Role: LEADER
I20260620 06:57:24.450860 1442 consensus_queue.cc:237] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } }
I20260620 06:57:24.454882 962 catalog_manager.cc:5495] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 0745651fee8e4a3e82fa2e18a5b3a3ea (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73936a5b2a844898bb5491fa9721baa0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3824ca1d95ae486aa8b8a22e465f4588" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0745651fee8e4a3e82fa2e18a5b3a3ea" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36733 } health_report { overall_health: HEALTHY } } }
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:24.479756 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.465782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36376 (local address 127.0.97.1:36733)
0620 06:57:24.465938 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.465942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.466449 (+ 507us) server_negotiation.cc:407] Connection header received
0620 06:57:24.466656 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.466660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.466735 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.466818 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.468235 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.468921 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.470509 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.470950 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.471059 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.474619 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.474639 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.474654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.474687 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.477184 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.477817 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.477825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.477827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.477976 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.478318 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.478323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.478325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.478694 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.478973 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.479224 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.479461 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:24.520687 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.507443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.97.1:36733)
0620 06:57:24.507664 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.507671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.507913 (+ 242us) server_negotiation.cc:407] Connection header received
0620 06:57:24.508100 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.508105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.508179 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.508260 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.509421 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.510146 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.511553 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.511965 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.512061 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.515933 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.515937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.515941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.515975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.518231 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.518821 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.518827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.518830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.518960 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.519370 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.519380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.519384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.519638 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.519902 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.520143 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.520356 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 06:57:24.545845 1419 tablet.cc:2369] T 2f4731edb25547bba06c6de5039036a3 P 3824ca1d95ae486aa8b8a22e465f4588: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260620 06:57:24.558265 1286 tablet.cc:2369] T 2f4731edb25547bba06c6de5039036a3 P 73936a5b2a844898bb5491fa9721baa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:57:24.562394 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.548581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.0.97.1:36733)
0620 06:57:24.548771 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.548778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.549095 (+ 317us) server_negotiation.cc:407] Connection header received
0620 06:57:24.549263 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.549267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.549353 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.549439 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.550880 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.551698 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.552887 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.553290 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.553396 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.556980 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.556984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.556988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.557021 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.560026 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.560633 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.560638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.560641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.560763 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.561129 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.561134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.561137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.561385 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.561608 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.561870 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.562082 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 06:57:24.564021 1066 tablet_service.cc:2908] Rejecting scan request for tablet 2f4731edb25547bba06c6de5039036a3: Uninitialized: clean time has not yet been initialized
I20260620 06:57:24.566778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1023
I20260620 06:57:24.578251 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.567550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.1:36733)
0620 06:57:24.567693 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.567697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.568137 (+ 440us) server_negotiation.cc:407] Connection header received
0620 06:57:24.568276 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.568280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.568335 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.568405 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.569455 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.570151 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.571373 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.571756 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.571857 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.573779 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.573783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.573787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.573814 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.576203 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.576758 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.576761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.576763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.576856 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.577121 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.577124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.577126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.577352 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.577516 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.577761 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.578005 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:57:24.578919 1146 generic_service.cc:196] Checking for leaks (request via RPC)
W20260620 06:57:24.667047 1152 tablet.cc:2369] T 2f4731edb25547bba06c6de5039036a3 P 0745651fee8e4a3e82fa2e18a5b3a3ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:57:24.667667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1023
I20260620 06:57:24.681295 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1157
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:24.696364 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.682123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.97.2:38811)
0620 06:57:24.682285 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.682289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.682306 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:57:24.682356 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.682359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.682434 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.682513 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.683614 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.684312 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.685587 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.685971 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.686066 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.690736 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.690750 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.690766 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.690802 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.693412 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.694171 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.694182 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.694186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.694355 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.694863 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.694868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.694872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.695280 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.695600 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.695827 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.696041 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:57:24.697016 1280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:24.782228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1157
I20260620 06:57:24.795071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1290
Jun 20 06:57:24 dist-test-slave-mz20 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:24.808645 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:24.795783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.0.97.3:33765)
0620 06:57:24.795940 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:24.795944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:24.795962 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:57:24.796108 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:24.796112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:24.796209 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:24.796307 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:24.797446 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.798147 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.799386 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:24.799769 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:24.799857 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:24.803644 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:24.803657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:24.803673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:24.803705 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:24.806250 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.806932 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.806940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.806942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.807073 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:24.807460 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:24.807464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:24.807466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:24.807826 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:24.808115 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:24.808244 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 06:57:24.808397 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:57:24.809275 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:24.896493 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1290
I20260620 06:57:24.909132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 930
I20260620 06:57:24.910226 992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:25.013866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 930
2026-06-20T06:57:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3796 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:25 dist-test-slave-mz20 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:25 dist-test-slave-mz20 krb5kdc[1466](info): set up 2 sockets
Jun 20 06:57:25 dist-test-slave-mz20 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 06:57:26.002259 388 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:26 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:26.168120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46529
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:26.471458 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:26.471730 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:26.471858 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:26.478924 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:26.479032 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:57:26.479094 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:26.479180 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:26.479249 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:26.479322 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:26.489146 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46529
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:26.490402 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:26.491693 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:26.502579 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:26.502485 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:26.502530 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:26.503381 1479 server_base.cc:1034] running on GCE node
I20260620 06:57:26.504065 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:26.505352 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:26.506572 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781938646506488 us; error 37 us; skew 500 ppm
I20260620 06:57:26.506882 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:26 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:26.513319 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:26.514822 1479 webserver.cc:460] Webserver started at http://127.0.97.62:41591/ using document root <none> and password file <none>
I20260620 06:57:26.515497 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:26.515586 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:26.515889 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:26.518759 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3"
format_stamp: "Formatted at 2026-06-20 06:57:26 on dist-test-slave-mz20"
server_key: "a64461874c3f608e7d27da85a0e958e8"
server_key_iv: "da8f64d7842d6b6dee6ab620e50142d5"
server_key_version: "encryptionkey@0"
I20260620 06:57:26.519536 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3"
format_stamp: "Formatted at 2026-06-20 06:57:26 on dist-test-slave-mz20"
server_key: "a64461874c3f608e7d27da85a0e958e8"
server_key_iv: "da8f64d7842d6b6dee6ab620e50142d5"
server_key_version: "encryptionkey@0"
I20260620 06:57:26.524523 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:57:26.527776 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:26.529064 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:26.529220 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3"
format_stamp: "Formatted at 2026-06-20 06:57:26 on dist-test-slave-mz20"
server_key: "a64461874c3f608e7d27da85a0e958e8"
server_key_iv: "da8f64d7842d6b6dee6ab620e50142d5"
server_key_version: "encryptionkey@0"
I20260620 06:57:26.529413 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:26.545554 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:26.549671 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:26.549921 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:26.569733 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46529
I20260620 06:57:26.569756 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46529 every 8 connection(s)
I20260620 06:57:26.571027 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:26.573052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1479
I20260620 06:57:26.573225 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:26.573833 388 external_mini_cluster.cc:1428] Setting key 8c6e4bad66154aa4570df0af8ac372c2
I20260620 06:57:26.574751 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 06:57:26 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:26.586024 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Bootstrap starting.
I20260620 06:57:26.589570 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:26.590746 1556 log.cc:824] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:26.593892 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: No bootstrap required, opened a new log
I20260620 06:57:26.594321 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:26.575583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.0.97.62:46529)
0620 06:57:26.576285 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:26.576305 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:26.576343 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:57:26.577852 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:26.577873 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:26.578248 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:26.578640 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:26.579973 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:26.581258 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:26.582428 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:26.583147 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:26.583341 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:26.587221 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:26.587229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:26.587243 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:26.587277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:26.590233 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:26.591049 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:26.591056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:26.591062 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:26.591220 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:26.591652 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:26.591657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:26.591660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:26.592252 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:26.592462 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:26.592940 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 06:57:26.593290 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":320,"threads_started":1}
I20260620 06:57:26.601085 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260620 06:57:26.601415 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:26.601506 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6fefbed8d04e1f8a93953bef2d8cc3, State: Initialized, Role: FOLLOWER
I20260620 06:57:26.602128 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260620 06:57:26.602332 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:26.602468 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:26.602648 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:26.604622 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260620 06:57:26.605086 1556 leader_election.cc:302] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6fefbed8d04e1f8a93953bef2d8cc3; no voters:
I20260620 06:57:26.605582 1556 leader_election.cc:288] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:26.605701 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:26.606685 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [term 1 LEADER]: Becoming Leader. State: Replica: ff6fefbed8d04e1f8a93953bef2d8cc3, State: Running, Role: LEADER
I20260620 06:57:26.607254 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } }
I20260620 06:57:26.607540 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:26.609721 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6fefbed8d04e1f8a93953bef2d8cc3. Latest consensus state: current_term: 1 leader_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } } }
I20260620 06:57:26.610093 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:26.610509 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6fefbed8d04e1f8a93953bef2d8cc3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46529 } } }
I20260620 06:57:26.610781 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:26.613849 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:26.619058 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:26.627722 1569 catalog_manager.cc:1347] Generated new cluster ID: 861da4c709d740598e65fdbb13b09d33
I20260620 06:57:26.627831 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:26.645277 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:26.646590 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:26.655848 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Generated new TSK 0
I20260620 06:57:26.657011 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:26.710590 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:27.019642 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:27.019956 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:27.020087 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:27.027275 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:27.027408 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:27.027535 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:27.027647 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:27.037595 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:27.038936 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:27.040289 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:27.053148 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:27.053124 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:27.053606 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:27.054705 1582 server_base.cc:1034] running on GCE node
I20260620 06:57:27.055398 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:27.056348 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:27.057572 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781938647057504 us; error 47 us; skew 500 ppm
I20260620 06:57:27.057868 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:27 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:27.061136 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:27.062561 1582 webserver.cc:460] Webserver started at http://127.0.97.1:44547/ using document root <none> and password file <none>
I20260620 06:57:27.063256 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:27.063341 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:27.063608 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:27.066489 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "ea5773c50f1b4e839fb6b940b1a36046"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "97824b0bbc57a1e0c5c6b36c7e417654"
server_key_iv: "105fbd73a7fc5946ad9de7de57d65b02"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.067324 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ea5773c50f1b4e839fb6b940b1a36046"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "97824b0bbc57a1e0c5c6b36c7e417654"
server_key_iv: "105fbd73a7fc5946ad9de7de57d65b02"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.072316 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:27.075749 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:27.077198 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 06:57:27.077351 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "ea5773c50f1b4e839fb6b940b1a36046"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "97824b0bbc57a1e0c5c6b36c7e417654"
server_key_iv: "105fbd73a7fc5946ad9de7de57d65b02"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.077587 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:27.097128 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:27.101343 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:27.101586 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:27.104192 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:27.104280 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:27.104418 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:27.104524 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 06:57:27 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:27.127475 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.104648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37789 (local address 127.0.97.62:46529)
0620 06:57:27.107280 (+ 2632us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:27.107285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:27.107304 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:57:27.109279 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:27.109286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:27.109366 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:27.109466 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:27.111723 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.112524 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.115462 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.115899 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.116032 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.120651 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:27.120657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:27.120660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:27.120695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:27.123564 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.124245 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.124251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.124254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.124398 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.125003 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.125008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.125012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.125333 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:27.126653 (+ 1320us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:27.126814 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 06:57:27.127064 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2522,"thread_start_us":2405,"threads_started":1}
I20260620 06:57:27.128077 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.105337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.1:37789)
0620 06:57:27.106274 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:27.106329 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:27.108947 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:27.109743 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:27.109757 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:27.110640 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:27.111524 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.111553 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.112672 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.112678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.115275 (+ 2597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.115291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.116156 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.116164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.116479 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.117220 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:27.117255 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:27.120369 (+ 3114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:27.123686 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.123695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.123712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.124119 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.124561 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.124566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.124570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.124838 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.125491 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:27.125500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:27.125885 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 06:57:27.126238 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 06:57:27.126847 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":376,"threads_started":1}
I20260620 06:57:27.147054 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.133318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38653 (local address 127.0.97.62:46529)
0620 06:57:27.133480 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:27.133486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:27.134011 (+ 525us) server_negotiation.cc:407] Connection header received
0620 06:57:27.134157 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:27.134163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:27.134240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:27.134344 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:27.135411 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.136281 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.138184 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.138741 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.138854 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.141083 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:27.141089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:27.141094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:27.141129 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:27.144184 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.144987 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.144991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.144993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.145121 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.145540 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.145545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.145548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.145828 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:27.145956 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:27.146489 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 06:57:27.146713 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:57:27.147233 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.133311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.1:38653)
0620 06:57:27.133924 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:27.133945 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:27.134050 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:27.134353 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:27.134356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:27.134822 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:27.135269 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.135285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.136401 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.136406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.138007 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.138021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.139074 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.139080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.139251 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.139871 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:27.139894 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:27.140899 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:27.144308 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.144313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.144317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.144835 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.145228 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.145234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.145239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.145412 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.146016 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:27.146021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:27.146250 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 06:57:27.146405 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 06:57:27.146773 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":324,"threads_started":1}
I20260620 06:57:27.152271 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32861
I20260620 06:57:27.152311 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32861 every 8 connection(s)
I20260620 06:57:27.153368 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:27.155107 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260620 06:57:27.155296 1718 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:27.155939 1718 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260620 06:57:27.157840 1511 ts_manager.cc:194] Registered new tserver with Master: ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1:32861)
I20260620 06:57:27.157970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1582
I20260620 06:57:27.158151 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:27.158931 388 external_mini_cluster.cc:1428] Setting key bda86121967d8bcaefec9946546b5c7e
I20260620 06:57:27.159735 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37789
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:27.195348 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:27.500352 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:27.500624 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:27.500741 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:27.507787 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:27.507884 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:27.508008 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:27.508117 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:27.517851 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:27.519109 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:27.520342 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:27.532370 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:27.532402 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:27.532464 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:27.533140 1722 server_base.cc:1034] running on GCE node
I20260620 06:57:27.533700 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:27.534564 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:27.535771 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781938647535702 us; error 50 us; skew 500 ppm
I20260620 06:57:27.536046 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:27 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:27.538873 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:27.540165 1722 webserver.cc:460] Webserver started at http://127.0.97.2:46827/ using document root <none> and password file <none>
I20260620 06:57:27.540772 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:27.540851 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:27.541098 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:27.543781 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "630d74098af24ecb9a7b9c33854f5cd3"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "4afcd852f1eb4e2a9e889a9e6067caca"
server_key_iv: "edf74b8c1c65cfa7343e5903312027fd"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.544492 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "630d74098af24ecb9a7b9c33854f5cd3"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "4afcd852f1eb4e2a9e889a9e6067caca"
server_key_iv: "edf74b8c1c65cfa7343e5903312027fd"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.549393 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:27.552687 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:27.553978 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:27.554128 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "630d74098af24ecb9a7b9c33854f5cd3"
format_stamp: "Formatted at 2026-06-20 06:57:27 on dist-test-slave-mz20"
server_key: "4afcd852f1eb4e2a9e889a9e6067caca"
server_key_iv: "edf74b8c1c65cfa7343e5903312027fd"
server_key_version: "encryptionkey@0"
I20260620 06:57:27.554317 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:27.581015 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:27.585100 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:27.585345 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:27.588020 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:27.588104 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:27.588230 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:27.588343 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 06:57:27 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:27.610230 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.588447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59083 (local address 127.0.97.62:46529)
0620 06:57:27.588628 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:27.588634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:27.590049 (+ 1415us) server_negotiation.cc:407] Connection header received
0620 06:57:27.592287 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:27.592294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:27.592379 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:27.592474 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:27.595115 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.596273 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.598984 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.599405 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.599516 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.604328 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:27.604333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:27.604336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:27.604370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:27.606774 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.607731 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.607739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.607743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.607901 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.608308 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.608313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.608316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.608712 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:27.608869 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:27.609578 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 06:57:27.609835 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:57:27.611254 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.589008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.2:59083)
0620 06:57:27.589858 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:27.589917 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:27.592001 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:27.592851 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:27.592864 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:27.593767 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:27.594875 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.594904 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.596469 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.596474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.598808 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.598825 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.599687 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.599695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.600064 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.600837 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:27.600886 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:27.604085 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:27.606960 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.606970 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.606989 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.607584 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.608004 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.608008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.608010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.608200 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.608884 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:27.608890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:27.609229 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 06:57:27.609458 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:57:27.609898 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260620 06:57:27.630592 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.616804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37385 (local address 127.0.97.62:46529)
0620 06:57:27.616968 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:27.616973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:27.617585 (+ 612us) server_negotiation.cc:407] Connection header received
0620 06:57:27.617730 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:27.617736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:27.617819 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:27.617912 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:27.618798 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.619773 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.621856 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.622233 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.622323 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.624687 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:27.624694 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:27.624699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:27.624735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:27.627754 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.628605 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.628612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.628615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.628782 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:27.629166 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:27.629171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:27.629175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:27.629532 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:27.629680 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:27.630136 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 06:57:27.630297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:57:27.630960 1802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:27.616929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.2:37385)
0620 06:57:27.617495 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:27.617517 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:27.617626 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:27.617921 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:27.617924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:27.618341 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:27.618664 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.618675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.619918 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.619924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.621705 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:27.621722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:27.622408 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:27.622415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:27.622637 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:27.623265 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:27.623287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:27.624499 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:27.627886 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.627894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.627898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.628467 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.628902 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:27.628907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:27.628910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:27.629064 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:27.629687 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:27.629693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:27.629913 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 06:57:27.630070 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 06:57:27.630469 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":314,"threads_started":1}
I20260620 06:57:27.642030 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42285
I20260620 06:57:27.642066 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42285 every 8 connection(s)
I20260620 06:57:27.643290 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:27.645146 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260620 06:57:27.645344 1857 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:27.645967 1857 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260620 06:57:27.647269 1510 ts_manager.cc:194] Registered new tserver with Master: 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285)
I20260620 06:57:27.648124 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59083
I20260620 06:57:27.651748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1722
I20260620 06:57:27.651880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:27.652282 388 external_mini_cluster.cc:1428] Setting key 60d6f278dbc16400b4a2b0b44a4de0e0
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:27.686040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46529
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:27.993505 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:27.993819 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:27.993950 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:28.001116 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:28.001242 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:28.001366 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:28.001473 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:28.011147 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:28.012375 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:28.013702 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:28.026194 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:28.026150 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:28.026230 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:28.026922 1861 server_base.cc:1034] running on GCE node
I20260620 06:57:28.027524 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:28.028364 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:28.029567 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781938648029501 us; error 39 us; skew 500 ppm
I20260620 06:57:28.029877 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:28 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:28.033028 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:28.034340 1861 webserver.cc:460] Webserver started at http://127.0.97.3:45747/ using document root <none> and password file <none>
I20260620 06:57:28.035022 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:28.035117 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:28.035404 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:28.038136 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "43cb298df9594fe5a6695c35d7a3e17e"
format_stamp: "Formatted at 2026-06-20 06:57:28 on dist-test-slave-mz20"
server_key: "22144c10d9d9eda69900d8df3da75c1a"
server_key_iv: "f21f921a59a87d9d2121f60171b87804"
server_key_version: "encryptionkey@0"
I20260620 06:57:28.038972 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "43cb298df9594fe5a6695c35d7a3e17e"
format_stamp: "Formatted at 2026-06-20 06:57:28 on dist-test-slave-mz20"
server_key: "22144c10d9d9eda69900d8df3da75c1a"
server_key_iv: "f21f921a59a87d9d2121f60171b87804"
server_key_version: "encryptionkey@0"
I20260620 06:57:28.043768 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 06:57:28.047010 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:28.048339 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 06:57:28.048497 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "43cb298df9594fe5a6695c35d7a3e17e"
format_stamp: "Formatted at 2026-06-20 06:57:28 on dist-test-slave-mz20"
server_key: "22144c10d9d9eda69900d8df3da75c1a"
server_key_iv: "f21f921a59a87d9d2121f60171b87804"
server_key_version: "encryptionkey@0"
I20260620 06:57:28.048689 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:28.061491 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:28.065824 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:28.066095 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:28.068751 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:28.068856 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:28.068981 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:28.069044 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:57:28 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:28.091902 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.069535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46755 (local address 127.0.97.62:46529)
0620 06:57:28.069735 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.069740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.071255 (+ 1515us) server_negotiation.cc:407] Connection header received
0620 06:57:28.073318 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.073325 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.073410 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.073500 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:28.075793 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.076776 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.079127 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.079665 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.079787 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.084807 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:28.084814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:28.084819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:28.084863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:28.088113 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.089086 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.089092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.089096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.089226 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.089771 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.089777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.089780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.090213 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:28.090348 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.091320 (+ 972us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.091543 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 06:57:28.093546 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.070285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.3:46755)
0620 06:57:28.071085 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.071130 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.073017 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.073748 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.073757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.074493 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:28.075557 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.075589 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.076917 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.076925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.078918 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.078938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.079901 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.079908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.080218 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.080986 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:28.081020 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:28.084499 (+ 3479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:28.088280 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.088288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.088302 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.088949 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.089334 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.089339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.089343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.089653 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.090348 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:28.090357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:28.090883 (+ 526us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.091207 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.091803 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":343,"threads_started":1}
I20260620 06:57:28.113982 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.099169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59403 (local address 127.0.97.62:46529)
0620 06:57:28.099365 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.099370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.099942 (+ 572us) server_negotiation.cc:407] Connection header received
0620 06:57:28.100194 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.100201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.100296 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.100411 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:28.101992 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.103046 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.105132 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.105661 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.105792 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.108118 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:28.108125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:28.108128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:28.108162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:28.110768 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.111620 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.111626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.111630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.111788 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.112319 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.112325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.112329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.112715 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:28.112867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.113359 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.113572 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:57:28.114034 1959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.099141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.97.3:59403)
0620 06:57:28.099810 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.099835 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.100005 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.100623 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.100632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.101258 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:28.101834 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.101848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.103218 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.103224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.104952 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.104974 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.105868 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.105876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.106062 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.106705 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:28.106723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:28.107897 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:28.110922 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.110929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.110937 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.111492 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.111924 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.111929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.111933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.112154 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.112850 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:28.112855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:28.113072 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.113218 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.113563 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":339,"threads_started":1}
I20260620 06:57:28.116786 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44043
I20260620 06:57:28.116786 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44043 every 8 connection(s)
I20260620 06:57:28.118300 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:28.120301 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46529
I20260620 06:57:28.120518 1996 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:28.121240 1996 heartbeater.cc:511] Master 127.0.97.62:46529 requested a full tablet report, sending...
I20260620 06:57:28.121248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 1861
I20260620 06:57:28.121390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:28.121924 388 external_mini_cluster.cc:1428] Setting key 083e663af3f3c78cb32af2f5178d7630
I20260620 06:57:28.122781 1510 ts_manager.cc:194] Registered new tserver with Master: 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043)
I20260620 06:57:28.123961 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46755
I20260620 06:57:28.124723 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:57:28 dist-test-slave-mz20 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:57:28 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:28.162231 1718 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
I20260620 06:57:28.166204 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.153157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.97.62:46529)
0620 06:57:28.153342 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.153348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.153648 (+ 300us) server_negotiation.cc:407] Connection header received
0620 06:57:28.153815 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.153821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.153898 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.153978 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:28.155330 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.156039 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.157268 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.157758 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.157861 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.161382 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:28.161386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:28.161390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:28.161420 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:28.163929 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.164478 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.164483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.164487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.164609 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.164928 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.164933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.164937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.165176 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:28.165395 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.165678 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.165890 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:57:28.169957 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:28.176517 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:28.210029 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.188277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44043 (local address 127.0.0.1:56208)
0620 06:57:28.201350 (+ 13073us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.201408 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.202387 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.203640 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.203647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.203675 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.204117 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.204129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.205562 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.205566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.207379 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.207392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.209122 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.209128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.209318 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.209367 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.209489 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.209675 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12745,"thread_start_us":15258,"threads_started":1}
I20260620 06:57:28.210752 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.188848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.97.3:44043)
0620 06:57:28.189605 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.189615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.201544 (+ 11929us) server_negotiation.cc:407] Connection header received
0620 06:57:28.202689 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.202699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.203123 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.203315 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.204349 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.205416 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.207556 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.208931 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.209041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.209273 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.210267 (+ 994us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.210425 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":521,"threads_started":1}
I20260620 06:57:28.214452 1926 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba9819bd3489495a1913cc9231481 (DEFAULT_TABLE table=test-table [id=25263f2625d54b46b68a9f782eccdc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.215250 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.188404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32861 (local address 127.0.0.1:53360)
0620 06:57:28.203337 (+ 14933us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.203360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.203458 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.208463 (+ 5005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.208471 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.208514 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.209052 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.209064 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.210812 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.210817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.212911 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.212925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.214518 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.214525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.214698 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.214729 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.214813 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.214915 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14817,"mutex_wait_us":56,"spinlock_wait_cycles":5504,"thread_start_us":13182,"threads_started":1}
I20260620 06:57:28.215912 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.189265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.0.97.1:32861)
0620 06:57:28.207921 (+ 18656us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.207929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.207953 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:28.208029 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.208035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.208244 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.208586 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.209245 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.210673 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.213087 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.214344 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.214454 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.214699 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.215468 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.215638 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18474,"thread_start_us":18355,"threads_started":1}
I20260620 06:57:28.216094 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba9819bd3489495a1913cc9231481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.216631 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.190420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42285 (local address 127.0.0.1:41168)
0620 06:57:28.207382 (+ 16962us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.207404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.207518 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.208295 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.208301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.208327 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.208770 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.208782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.210704 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.210709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.213604 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.213618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.215885 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.215891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.216054 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.216097 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.216179 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.216297 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16832,"mutex_wait_us":44,"thread_start_us":16700,"threads_started":1}
I20260620 06:57:28.217272 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.190912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.0.97.2:42285)
0620 06:57:28.206972 (+ 16060us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.206981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.207680 (+ 699us) server_negotiation.cc:407] Connection header received
0620 06:57:28.207780 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.207788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.208097 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.208287 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.208973 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.210586 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.213764 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.215308 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.215553 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.215738 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.216836 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.216996 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15895,"thread_start_us":15773,"threads_started":1}
I20260620 06:57:28.219384 1648 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba9819bd3489495a1913cc9231481 (DEFAULT_TABLE table=test-table [id=25263f2625d54b46b68a9f782eccdc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.220779 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba9819bd3489495a1913cc9231481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.221324 1787 tablet_service.cc:1467] Processing CreateTablet for tablet e2dba9819bd3489495a1913cc9231481 (DEFAULT_TABLE table=test-table [id=25263f2625d54b46b68a9f782eccdc21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.222779 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2dba9819bd3489495a1913cc9231481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.227236 2015 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Bootstrap starting.
I20260620 06:57:28.230252 2015 tablet_bootstrap.cc:654] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.231292 2015 log.cc:824] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:28.233749 2015 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: No bootstrap required, opened a new log
I20260620 06:57:28.233982 2015 ts_tablet_manager.cc:1397] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:57:28.239315 2017 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Bootstrap starting.
I20260620 06:57:28.242769 2017 tablet_bootstrap.cc:654] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.242580 2015 raft_consensus.cc:348] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.243003 2015 raft_consensus.cc:374] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.243105 2015 raft_consensus.cc:729] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cb298df9594fe5a6695c35d7a3e17e, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.243177 2018 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Bootstrap starting.
I20260620 06:57:28.244273 2017 log.cc:824] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:28.244148 2015 consensus_queue.cc:260] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.245237 1996 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
I20260620 06:57:28.245955 2018 tablet_bootstrap.cc:654] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.247365 2015 ts_tablet_manager.cc:1428] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 06:57:28.247627 2018 log.cc:824] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:28.250334 2017 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: No bootstrap required, opened a new log
I20260620 06:57:28.250591 2018 tablet_bootstrap.cc:492] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: No bootstrap required, opened a new log
I20260620 06:57:28.250648 2017 ts_tablet_manager.cc:1397] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 06:57:28.250836 2018 ts_tablet_manager.cc:1397] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 06:57:28.254993 2019 raft_consensus.cc:482] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:28.255276 2019 raft_consensus.cc:504] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.258086 2019 leader_election.cc:288] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1:32861), 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285)
I20260620 06:57:28.257915 2017 raft_consensus.cc:348] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.258329 2017 raft_consensus.cc:374] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.258417 2017 raft_consensus.cc:729] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5773c50f1b4e839fb6b940b1a36046, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.259063 2017 consensus_queue.cc:260] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.259706 2018 raft_consensus.cc:348] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.260072 2018 raft_consensus.cc:374] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.260169 2018 raft_consensus.cc:729] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630d74098af24ecb9a7b9c33854f5cd3, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.260695 2018 consensus_queue.cc:260] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.265021 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.257553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42285 (local address 127.0.97.3:48699)
0620 06:57:28.257829 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.257847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.257952 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.258833 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.258839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.258864 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.259032 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.259043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.260366 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.260370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.262355 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.262371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.264235 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.264242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.264443 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.264480 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.264564 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.264718 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260620 06:57:28.265699 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.258206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48699 (local address 127.0.97.2:42285)
0620 06:57:28.258384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.258390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.258408 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:57:28.258471 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.258476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.258571 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.258668 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.259177 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.260222 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.262564 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.263903 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.264033 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.264148 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.265289 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.265432 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:57:28.266589 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba9819bd3489495a1913cc9231481" candidate_uuid: "43cb298df9594fe5a6695c35d7a3e17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3" is_pre_election: true
I20260620 06:57:28.267104 1812 raft_consensus.cc:2454] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43cb298df9594fe5a6695c35d7a3e17e in term 0.
I20260620 06:57:28.267896 1879 leader_election.cc:302] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cb298df9594fe5a6695c35d7a3e17e, 630d74098af24ecb9a7b9c33854f5cd3; no voters:
I20260620 06:57:28.268393 2019 raft_consensus.cc:2781] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:28.268591 2019 raft_consensus.cc:482] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:28.268710 2017 ts_tablet_manager.cc:1428] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260620 06:57:28.268733 2019 raft_consensus.cc:3037] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.269765 1857 heartbeater.cc:503] Master 127.0.97.62:46529 was elected leader, sending a full tablet report...
I20260620 06:57:28.270329 2018 ts_tablet_manager.cc:1428] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260620 06:57:28.272154 2019 raft_consensus.cc:504] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.273371 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba9819bd3489495a1913cc9231481" candidate_uuid: "43cb298df9594fe5a6695c35d7a3e17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3"
I20260620 06:57:28.273566 1812 raft_consensus.cc:3037] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.274641 2019 leader_election.cc:288] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [CANDIDATE]: Term 1 election: Requested vote from peers ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1:32861), 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285)
I20260620 06:57:28.276062 1812 raft_consensus.cc:2454] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43cb298df9594fe5a6695c35d7a3e17e in term 1.
I20260620 06:57:28.276311 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.257771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36377 (local address 127.0.97.1:32861)
0620 06:57:28.257937 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.257942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.267201 (+ 9259us) server_negotiation.cc:407] Connection header received
0620 06:57:28.267268 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.267273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.267850 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.267936 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.269518 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.270501 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.274635 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.275428 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.275510 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.275590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.275968 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.276099 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:57:28.276350 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.258673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32861 (local address 127.0.97.3:36377)
0620 06:57:28.266301 (+ 7628us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.266320 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.266429 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.269007 (+ 2578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.269012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.269056 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.269230 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.269241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.271848 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.271852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.273626 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.273641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.275699 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.275703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.275822 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.275843 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.275916 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.276049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7525,"thread_start_us":15649,"threads_started":1}
I20260620 06:57:28.276422 1879 leader_election.cc:302] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cb298df9594fe5a6695c35d7a3e17e, 630d74098af24ecb9a7b9c33854f5cd3; no voters:
I20260620 06:57:28.276700 2019 raft_consensus.cc:2781] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:28.277217 2019 raft_consensus.cc:686] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 1 LEADER]: Becoming Leader. State: Replica: 43cb298df9594fe5a6695c35d7a3e17e, State: Running, Role: LEADER
I20260620 06:57:28.277781 2019 consensus_queue.cc:237] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } }
I20260620 06:57:28.277542 1672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba9819bd3489495a1913cc9231481" candidate_uuid: "43cb298df9594fe5a6695c35d7a3e17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5773c50f1b4e839fb6b940b1a36046"
I20260620 06:57:28.277513 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2dba9819bd3489495a1913cc9231481" candidate_uuid: "43cb298df9594fe5a6695c35d7a3e17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea5773c50f1b4e839fb6b940b1a36046" is_pre_election: true
I20260620 06:57:28.278625 1672 raft_consensus.cc:3037] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.281399 1672 raft_consensus.cc:2454] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43cb298df9594fe5a6695c35d7a3e17e in term 1.
I20260620 06:57:28.282801 1510 catalog_manager.cc:5495] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e reported cstate change: term changed from 0 to 1, leader changed from <none> to 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "43cb298df9594fe5a6695c35d7a3e17e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } health_report { overall_health: UNKNOWN } } }
I20260620 06:57:28.304842 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.299709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.0.97.3:44043)
0620 06:57:28.299871 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.299876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.300001 (+ 125us) server_negotiation.cc:407] Connection header received
0620 06:57:28.300250 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.300255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.300335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.300497 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:28.300857 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.301631 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.302940 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.303356 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.303457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.303859 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:28.304350 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:28.304450 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.304548 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.304615 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:57:28.313042 1672 raft_consensus.cc:1261] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Refusing update from remote peer 43cb298df9594fe5a6695c35d7a3e17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:28.313090 1812 raft_consensus.cc:1261] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Refusing update from remote peer 43cb298df9594fe5a6695c35d7a3e17e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:28.313879 2025 consensus_queue.cc:1035] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:28.314208 2019 consensus_queue.cc:1035] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [LEADER]: Connected to new peer: Peer: permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:28.327320 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298820703480860672 to 7298820703351062528. Current Snapshot: MvccSnapshot[applied={T|T < 7298820703480860672}]
I20260620 06:57:28.343990 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:28.344210 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:28.347415 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Sending DeleteTablet for 3 replicas of tablet e2dba9819bd3489495a1913cc9231481
I20260620 06:57:28.348412 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba9819bd3489495a1913cc9231481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56208
I20260620 06:57:28.348569 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba9819bd3489495a1913cc9231481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53360
I20260620 06:57:28.349232 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet e2dba9819bd3489495a1913cc9231481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41168
I20260620 06:57:28.349956 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:28.350625 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:28.353343 2039 tablet_replica.cc:330] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: stopping tablet replica
I20260620 06:57:28.353807 2039 raft_consensus.cc:2229] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:28.354303 2039 raft_consensus.cc:2258] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:28.355551 2038 tablet_replica.cc:330] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: stopping tablet replica
I20260620 06:57:28.355959 2038 raft_consensus.cc:2229] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:28.356418 2038 raft_consensus.cc:2258] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:28.358455 2039 ts_tablet_manager.cc:1905] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:28.359426 1925 tablet_service.cc:1467] Processing CreateTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e (DEFAULT_TABLE table=test-table [id=6fbc8a8d3307429096523d431c34cddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.359970 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d4bc99bfc44fb69f5e62f6515e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.365504 2039 ts_tablet_manager.cc:1918] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:28.365670 2039 log.cc:1196] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/e2dba9819bd3489495a1913cc9231481
I20260620 06:57:28.366209 2039 ts_tablet_manager.cc:1939] T e2dba9819bd3489495a1913cc9231481 P 630d74098af24ecb9a7b9c33854f5cd3: Deleting consensus metadata
I20260620 06:57:28.367275 1648 tablet_service.cc:1467] Processing CreateTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e (DEFAULT_TABLE table=test-table [id=6fbc8a8d3307429096523d431c34cddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.367727 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d4bc99bfc44fb69f5e62f6515e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.368511 2038 ts_tablet_manager.cc:1905] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:28.368685 1787 tablet_service.cc:1467] Processing CreateTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e (DEFAULT_TABLE table=test-table [id=6fbc8a8d3307429096523d431c34cddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:28.369314 2015 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Bootstrap starting.
I20260620 06:57:28.369839 2037 tablet_replica.cc:330] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: stopping tablet replica
I20260620 06:57:28.370745 2037 raft_consensus.cc:2229] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:28.370951 2015 tablet_bootstrap.cc:654] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.371387 2037 raft_consensus.cc:2258] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:28.373276 2037 ts_tablet_manager.cc:1905] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:28.377017 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d4bc99bfc44fb69f5e62f6515e623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.378312 1497 catalog_manager.cc:4841] TS 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285): tablet e2dba9819bd3489495a1913cc9231481 (table test-table [id=25263f2625d54b46b68a9f782eccdc21]) successfully deleted
I20260620 06:57:28.379649 2037 ts_tablet_manager.cc:1918] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:28.379791 2037 log.cc:1196] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/e2dba9819bd3489495a1913cc9231481
I20260620 06:57:28.380275 2037 ts_tablet_manager.cc:1939] T e2dba9819bd3489495a1913cc9231481 P 43cb298df9594fe5a6695c35d7a3e17e: Deleting consensus metadata
I20260620 06:57:28.380267 2038 ts_tablet_manager.cc:1918] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:28.380498 2038 log.cc:1196] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/e2dba9819bd3489495a1913cc9231481
I20260620 06:57:28.381067 2038 ts_tablet_manager.cc:1939] T e2dba9819bd3489495a1913cc9231481 P ea5773c50f1b4e839fb6b940b1a36046: Deleting consensus metadata
I20260620 06:57:28.381141 2017 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Bootstrap starting.
I20260620 06:57:28.382833 1496 catalog_manager.cc:4841] TS 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043): tablet e2dba9819bd3489495a1913cc9231481 (table test-table [id=25263f2625d54b46b68a9f782eccdc21]) successfully deleted
I20260620 06:57:28.383134 2017 tablet_bootstrap.cc:654] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.383303 2018 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Bootstrap starting.
I20260620 06:57:28.384747 1498 catalog_manager.cc:4841] TS ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1:32861): tablet e2dba9819bd3489495a1913cc9231481 (table test-table [id=25263f2625d54b46b68a9f782eccdc21]) successfully deleted
I20260620 06:57:28.384848 2018 tablet_bootstrap.cc:654] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.385478 2015 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: No bootstrap required, opened a new log
I20260620 06:57:28.385665 2015 ts_tablet_manager.cc:1397] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260620 06:57:28.386572 2015 raft_consensus.cc:348] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.386920 2015 raft_consensus.cc:374] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.387072 2015 raft_consensus.cc:729] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cb298df9594fe5a6695c35d7a3e17e, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.387383 2015 consensus_queue.cc:260] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.389961 2015 ts_tablet_manager.cc:1428] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 06:57:28.393563 2017 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: No bootstrap required, opened a new log
I20260620 06:57:28.393633 2018 tablet_bootstrap.cc:492] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: No bootstrap required, opened a new log
I20260620 06:57:28.393714 2017 ts_tablet_manager.cc:1397] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260620 06:57:28.393787 2018 ts_tablet_manager.cc:1397] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 06:57:28.394361 2017 raft_consensus.cc:348] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.394529 2017 raft_consensus.cc:374] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.394649 2017 raft_consensus.cc:729] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5773c50f1b4e839fb6b940b1a36046, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.394939 2017 consensus_queue.cc:260] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.394899 2018 raft_consensus.cc:348] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.395112 2018 raft_consensus.cc:374] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.395195 2018 raft_consensus.cc:729] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630d74098af24ecb9a7b9c33854f5cd3, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.395406 2018 consensus_queue.cc:260] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.395540 2017 ts_tablet_manager.cc:1428] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:57:28.395999 2018 ts_tablet_manager.cc:1428] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260620 06:57:28.405488 1719 tablet.cc:2369] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:57:28.620663 1997 tablet.cc:2369] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:57:28.645188 1858 tablet.cc:2369] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:57:28.790947 2023 raft_consensus.cc:482] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:28.791141 2023 raft_consensus.cc:504] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.792622 2023 leader_election.cc:288] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285), 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043)
I20260620 06:57:28.799154 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.793302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44043 (local address 127.0.97.1:33183)
0620 06:57:28.793923 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.793946 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.794117 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.794456 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.794459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.794480 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.794787 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.794796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.795803 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.795809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.797424 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.797437 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.798526 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.798532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.798706 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.798730 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.798799 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.798884 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"spinlock_wait_cycles":7040,"thread_start_us":474,"threads_started":1}
I20260620 06:57:28.799247 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.793102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33183 (local address 127.0.97.3:44043)
0620 06:57:28.793288 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.793293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.794056 (+ 763us) server_negotiation.cc:407] Connection header received
0620 06:57:28.794257 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.794263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.794339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.794424 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.794908 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.795678 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.797609 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.798378 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.798473 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.798582 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.798840 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.798986 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 06:57:28.799993 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d4bc99bfc44fb69f5e62f6515e623e" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb298df9594fe5a6695c35d7a3e17e" is_pre_election: true
I20260620 06:57:28.800319 1946 raft_consensus.cc:2454] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 0.
I20260620 06:57:28.800541 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.793015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34701 (local address 127.0.97.2:42285)
0620 06:57:28.793704 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.793709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.794312 (+ 603us) server_negotiation.cc:407] Connection header received
0620 06:57:28.794950 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.794954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.795437 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.795536 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:28.796142 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.796914 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.798633 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.799672 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.799794 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.799887 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.800147 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.800272 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":372,"threads_started":1}
I20260620 06:57:28.800556 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.793510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42285 (local address 127.0.97.1:34701)
0620 06:57:28.794210 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.794226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.794345 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.795557 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.795562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.795585 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:28.795920 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.795930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.797065 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.797072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.798468 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.798482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.799860 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.799866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.800014 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.800039 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.800115 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.800235 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":449,"threads_started":1}
I20260620 06:57:28.800968 1599 leader_election.cc:302] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cb298df9594fe5a6695c35d7a3e17e, ea5773c50f1b4e839fb6b940b1a36046; no voters:
I20260620 06:57:28.801151 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d4bc99bfc44fb69f5e62f6515e623e" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3" is_pre_election: true
I20260620 06:57:28.801362 2023 raft_consensus.cc:2781] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:28.801390 1812 raft_consensus.cc:2454] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 0.
I20260620 06:57:28.801473 2023 raft_consensus.cc:482] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:28.801578 2023 raft_consensus.cc:3037] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.803802 2023 raft_consensus.cc:504] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.804347 2023 leader_election.cc:288] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 election: Requested vote from peers 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285), 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043)
I20260620 06:57:28.804760 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d4bc99bfc44fb69f5e62f6515e623e" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3"
I20260620 06:57:28.804806 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d4bc99bfc44fb69f5e62f6515e623e" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb298df9594fe5a6695c35d7a3e17e"
I20260620 06:57:28.804970 1812 raft_consensus.cc:3037] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.804972 1946 raft_consensus.cc:3037] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.807137 1812 raft_consensus.cc:2454] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 1.
I20260620 06:57:28.807482 1946 raft_consensus.cc:2454] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 1.
I20260620 06:57:28.807508 1600 leader_election.cc:302] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 630d74098af24ecb9a7b9c33854f5cd3, ea5773c50f1b4e839fb6b940b1a36046; no voters:
I20260620 06:57:28.807777 2023 raft_consensus.cc:2781] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:28.808205 2023 raft_consensus.cc:686] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 1 LEADER]: Becoming Leader. State: Replica: ea5773c50f1b4e839fb6b940b1a36046, State: Running, Role: LEADER
I20260620 06:57:28.808627 2023 consensus_queue.cc:237] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.811795 1510 catalog_manager.cc:5495] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:57:28 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:28.953433 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.940596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.0.97.62:46529)
0620 06:57:28.941155 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:28.941159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:28.941514 (+ 355us) server_negotiation.cc:407] Connection header received
0620 06:57:28.941738 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:28.941742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:28.941802 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:28.941893 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:28.942987 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.943720 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.944994 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.945420 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.945571 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.948657 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:28.948662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:28.948665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:28.948699 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:28.951126 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.951762 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.951765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.951767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.951872 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:28.952204 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:28.952208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:28.952210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:28.952434 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:28.952531 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:28.952915 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 06:57:28.953121 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":445,"thread_start_us":314,"threads_started":1}
I20260620 06:57:28.953472 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:28.940596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46529 (local address 127.0.0.1:35186)
0620 06:57:28.941398 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:28.941414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:28.941580 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:28.941972 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:28.941976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:28.942668 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:28.942846 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.942855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.943839 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.943843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.944846 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:28.944858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:28.945623 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:28.945628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:28.945763 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:28.946412 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:28.946429 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:28.948498 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:28.951258 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.951267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.951273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.951663 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.951951 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:28.951954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:28.951956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:28.952123 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:28.952544 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:28.952550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:28.952682 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 06:57:28.952819 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 06:57:28.953029 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":543,"threads_started":1}
I20260620 06:57:28.957914 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35186:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:57:28.958832 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:28.964648 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 6b10da91d17842c686a985a14217df9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f088866d1dc41c394591e05cfabbebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:28.964717 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 6b10da91d17842c686a985a14217df9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f088866d1dc41c394591e05cfabbebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:28.964859 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 6b10da91d17842c686a985a14217df9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f088866d1dc41c394591e05cfabbebe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:28.965133 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b10da91d17842c686a985a14217df9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.965243 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b10da91d17842c686a985a14217df9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.965286 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b10da91d17842c686a985a14217df9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:28.970583 2055 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: Bootstrap starting.
I20260620 06:57:28.972002 2057 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: Bootstrap starting.
I20260620 06:57:28.972293 2055 tablet_bootstrap.cc:654] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.972589 2056 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: Bootstrap starting.
I20260620 06:57:28.973934 2057 tablet_bootstrap.cc:654] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.974589 2056 tablet_bootstrap.cc:654] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:28.974924 2055 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: No bootstrap required, opened a new log
I20260620 06:57:28.975066 2055 ts_tablet_manager.cc:1397] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:57:28.975805 2055 raft_consensus.cc:348] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.976022 2055 raft_consensus.cc:374] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.976094 2055 raft_consensus.cc:729] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5773c50f1b4e839fb6b940b1a36046, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.976135 2057 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: No bootstrap required, opened a new log
I20260620 06:57:28.976258 2057 ts_tablet_manager.cc:1397] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 06:57:28.976271 2055 consensus_queue.cc:260] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.976663 2023 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:28.976935 2023 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:28.976962 2057 raft_consensus.cc:348] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.977191 2057 raft_consensus.cc:374] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.977289 2057 raft_consensus.cc:729] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 630d74098af24ecb9a7b9c33854f5cd3, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.977347 2055 ts_tablet_manager.cc:1428] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:57:28.977514 2057 consensus_queue.cc:260] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.978178 2056 tablet_bootstrap.cc:492] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: No bootstrap required, opened a new log
I20260620 06:57:28.978242 2057 ts_tablet_manager.cc:1428] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:57:28.978329 2056 ts_tablet_manager.cc:1397] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 06:57:28.978170 2061 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:28.978641 2061 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:28.979050 2056 raft_consensus.cc:348] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.979254 2056 raft_consensus.cc:374] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:28.979343 2056 raft_consensus.cc:729] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cb298df9594fe5a6695c35d7a3e17e, State: Initialized, Role: FOLLOWER
I20260620 06:57:28.979573 2056 consensus_queue.cc:260] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.981302 2056 ts_tablet_manager.cc:1428] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:28.981695 2062 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:28.981971 2062 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:28.985096 2023 raft_consensus.cc:482] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:28.985242 2023 raft_consensus.cc:504] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.985796 2023 leader_election.cc:288] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285), 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043)
I20260620 06:57:28.986176 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b10da91d17842c686a985a14217df9d" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3" is_pre_election: true
I20260620 06:57:28.986207 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b10da91d17842c686a985a14217df9d" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb298df9594fe5a6695c35d7a3e17e" is_pre_election: true
I20260620 06:57:28.986375 1946 raft_consensus.cc:2454] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 0.
I20260620 06:57:28.986380 1812 raft_consensus.cc:2454] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 0.
I20260620 06:57:28.986701 1599 leader_election.cc:302] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cb298df9594fe5a6695c35d7a3e17e, ea5773c50f1b4e839fb6b940b1a36046; no voters:
I20260620 06:57:28.987349 2023 raft_consensus.cc:2781] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:28.987437 2023 raft_consensus.cc:482] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:28.987538 2023 raft_consensus.cc:3037] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.989708 2023 raft_consensus.cc:504] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.990178 2023 leader_election.cc:288] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 election: Requested vote from peers 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285), 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043)
I20260620 06:57:28.990562 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b10da91d17842c686a985a14217df9d" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb298df9594fe5a6695c35d7a3e17e"
I20260620 06:57:28.990746 1946 raft_consensus.cc:3037] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.990763 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b10da91d17842c686a985a14217df9d" candidate_uuid: "ea5773c50f1b4e839fb6b940b1a36046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "630d74098af24ecb9a7b9c33854f5cd3"
I20260620 06:57:28.990976 1812 raft_consensus.cc:3037] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:28.992575 1946 raft_consensus.cc:2454] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 1.
I20260620 06:57:28.992816 1812 raft_consensus.cc:2454] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5773c50f1b4e839fb6b940b1a36046 in term 1.
I20260620 06:57:28.992941 1599 leader_election.cc:302] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43cb298df9594fe5a6695c35d7a3e17e, ea5773c50f1b4e839fb6b940b1a36046; no voters:
I20260620 06:57:28.993191 2023 raft_consensus.cc:2781] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:28.993328 2023 raft_consensus.cc:686] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [term 1 LEADER]: Becoming Leader. State: Replica: ea5773c50f1b4e839fb6b940b1a36046, State: Running, Role: LEADER
I20260620 06:57:28.993533 2023 consensus_queue.cc:237] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } }
I20260620 06:57:28.994933 2046 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: TxnStatusTablet state changed. Reason: New leader ea5773c50f1b4e839fb6b940b1a36046. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:28.995285 2046 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:28.995883 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:57:28.996122 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:57:28.996418 1510 catalog_manager.cc:5495] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260620 06:57:29.016577 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.009855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.0.97.1:32861)
0620 06:57:29.010422 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.010426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.010448 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:57:29.010526 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.010529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.010641 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.010764 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:29.011154 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.011935 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.014291 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.014796 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.014902 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.015427 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:29.015963 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:29.016066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.016242 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.016304 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":348,"threads_started":1}
I20260620 06:57:29.016646 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.009844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32861 (local address 127.0.0.1:42222)
0620 06:57:29.010080 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:29.010100 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:29.010246 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:29.010811 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:29.010816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:29.010828 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:29.011012 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:29.011023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.012052 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.012058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:29.014120 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:29.014143 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.014944 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.014951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:29.015146 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.015218 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:29.016065 (+ 847us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:29.016091 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:57:29.016181 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:57:29.016330 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:57:29.021222 1812 raft_consensus.cc:1261] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Refusing update from remote peer ea5773c50f1b4e839fb6b940b1a36046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:29.021478 1946 raft_consensus.cc:1261] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Refusing update from remote peer ea5773c50f1b4e839fb6b940b1a36046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:29.022022 2046 consensus_queue.cc:1035] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:29.022734 2046 consensus_queue.cc:1035] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:29.032495 2062 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: TxnStatusTablet state changed. Reason: New leader ea5773c50f1b4e839fb6b940b1a36046. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.032774 2062 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:29.033521 2068 mvcc.cc:205] Tried to move back new op lower bound from 7298820706383138816 to 7298820706279211008. Current Snapshot: MvccSnapshot[applied={T|T < 7298820706383138816}]
I20260620 06:57:29.034921 2062 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.035161 2062 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 43cb298df9594fe5a6695c35d7a3e17e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:29.036510 2046 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.036765 2046 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:29.048590 2061 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: TxnStatusTablet state changed. Reason: New leader ea5773c50f1b4e839fb6b940b1a36046. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.048931 2061 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:29.048918 2046 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.049198 2046 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P ea5773c50f1b4e839fb6b940b1a36046: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:29.052623 2061 tablet_replica.cc:439] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea5773c50f1b4e839fb6b940b1a36046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea5773c50f1b4e839fb6b940b1a36046" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32861 } } peers { permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 } } peers { permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 } } }
I20260620 06:57:29.052862 2061 tablet_replica.cc:442] T 6b10da91d17842c686a985a14217df9d P 630d74098af24ecb9a7b9c33854f5cd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:29.055433 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.046294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.97.1:32861)
0620 06:57:29.046444 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.046462 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.049060 (+ 2598us) server_negotiation.cc:407] Connection header received
0620 06:57:29.049136 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.049141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.049251 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.049334 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:29.050064 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.051269 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.053399 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.054052 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.054142 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.054752 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:29.054873 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:29.054959 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.055102 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.055176 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 06:57:29.071743 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.066179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37807 (local address 127.0.97.1:32861)
0620 06:57:29.066360 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.066367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.067022 (+ 655us) server_negotiation.cc:407] Connection header received
0620 06:57:29.067214 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.067217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.067270 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.067349 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:29.067865 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.068609 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.070031 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.070476 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.070613 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.071007 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:29.071102 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:29.071172 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.071283 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.071374 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":45}
I20260620 06:57:29.071815 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.066242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32861 (local address 127.0.97.1:37807)
0620 06:57:29.066921 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:29.066936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:29.067086 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:29.067363 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:29.067367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:29.067376 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:29.067722 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:29.067731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.068741 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.068746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:29.069909 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:29.069920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.070674 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.070680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:29.070790 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.070846 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:29.071186 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:29.071196 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 06:57:29.071262 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:57:29.071413 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":22,"thread_start_us":393,"threads_started":1}
I20260620 06:57:29.080626 1946 raft_consensus.cc:1261] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Refusing update from remote peer ea5773c50f1b4e839fb6b940b1a36046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:29.080765 1812 raft_consensus.cc:1261] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Refusing update from remote peer ea5773c50f1b4e839fb6b940b1a36046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:29.081307 2075 consensus_queue.cc:1035] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43cb298df9594fe5a6695c35d7a3e17e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:29.081601 2023 consensus_queue.cc:1035] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "630d74098af24ecb9a7b9c33854f5cd3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:29.139701 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:29.140285 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:29.143581 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff6fefbed8d04e1f8a93953bef2d8cc3: Sending DeleteTablet for 3 replicas of tablet b2d4bc99bfc44fb69f5e62f6515e623e
I20260620 06:57:29.144271 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53360
I20260620 06:57:29.144408 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56208
I20260620 06:57:29.144556 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet b2d4bc99bfc44fb69f5e62f6515e623e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41168
I20260620 06:57:29.156247 2086 tablet_replica.cc:330] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: stopping tablet replica
I20260620 06:57:29.156453 2086 raft_consensus.cc:2229] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:29.157047 2087 tablet_replica.cc:330] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: stopping tablet replica
I20260620 06:57:29.158003 2086 raft_consensus.cc:2258] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:29.157231 2087 raft_consensus.cc:2229] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:29.159226 2087 raft_consensus.cc:2258] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:29.159636 2086 ts_tablet_manager.cc:1905] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:29.160005 2085 tablet_replica.cc:330] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: stopping tablet replica
I20260620 06:57:29.160189 2085 raft_consensus.cc:2229] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:29.160766 2085 raft_consensus.cc:2258] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:29.165184 2085 ts_tablet_manager.cc:1905] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:29.166298 2086 ts_tablet_manager.cc:1918] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:29.166424 2086 log.cc:1196] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/b2d4bc99bfc44fb69f5e62f6515e623e
I20260620 06:57:29.166841 2086 ts_tablet_manager.cc:1939] T b2d4bc99bfc44fb69f5e62f6515e623e P 43cb298df9594fe5a6695c35d7a3e17e: Deleting consensus metadata
I20260620 06:57:29.167867 1496 catalog_manager.cc:4841] TS 43cb298df9594fe5a6695c35d7a3e17e (127.0.97.3:44043): tablet b2d4bc99bfc44fb69f5e62f6515e623e (table test-table [id=6fbc8a8d3307429096523d431c34cddd]) successfully deleted
I20260620 06:57:29.168255 2087 ts_tablet_manager.cc:1905] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:29.174798 2087 ts_tablet_manager.cc:1918] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:29.174917 2087 log.cc:1196] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/b2d4bc99bfc44fb69f5e62f6515e623e
I20260620 06:57:29.175333 2087 ts_tablet_manager.cc:1939] T b2d4bc99bfc44fb69f5e62f6515e623e P 630d74098af24ecb9a7b9c33854f5cd3: Deleting consensus metadata
I20260620 06:57:29.176301 1497 catalog_manager.cc:4841] TS 630d74098af24ecb9a7b9c33854f5cd3 (127.0.97.2:42285): tablet b2d4bc99bfc44fb69f5e62f6515e623e (table test-table [id=6fbc8a8d3307429096523d431c34cddd]) successfully deleted
I20260620 06:57:29.178131 2085 ts_tablet_manager.cc:1918] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:29.178277 2085 log.cc:1196] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/b2d4bc99bfc44fb69f5e62f6515e623e
I20260620 06:57:29.178718 2085 ts_tablet_manager.cc:1939] T b2d4bc99bfc44fb69f5e62f6515e623e P ea5773c50f1b4e839fb6b940b1a36046: Deleting consensus metadata
I20260620 06:57:29.179724 1498 catalog_manager.cc:4841] TS ea5773c50f1b4e839fb6b940b1a36046 (127.0.97.1:32861): tablet b2d4bc99bfc44fb69f5e62f6515e623e (table test-table [id=6fbc8a8d3307429096523d431c34cddd]) successfully deleted
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:29.190980 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.164116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.97.1:32861)
0620 06:57:29.164307 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.164313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.171768 (+ 7455us) server_negotiation.cc:407] Connection header received
0620 06:57:29.172255 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.172262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.172359 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.172427 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:29.174351 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.175562 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.177518 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.178218 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.178313 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.183918 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:29.183935 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:29.183954 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:29.183988 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:29.187783 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.188591 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.188600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.188604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.188753 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.189118 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.189123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.189126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.189597 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:29.189887 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.190176 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.190397 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 06:57:29.192157 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42242
I20260620 06:57:29.217224 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.203760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.97.62:46529)
0620 06:57:29.203990 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.203996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.204165 (+ 169us) server_negotiation.cc:407] Connection header received
0620 06:57:29.204346 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.204351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.204428 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.204535 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:29.206046 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.207201 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.208765 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.209421 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.209548 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.211581 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:29.211588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:29.211593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:29.211630 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:29.214843 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.215455 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.215461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.215465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.215604 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.215924 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.215928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.215932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.216260 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:29.216488 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.216690 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.216898 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 06:57:29.217756 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35192
I20260620 06:57:29.219698 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1582
I20260620 06:57:29.232378 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.220742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.0.97.1:32861)
0620 06:57:29.220908 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.220913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.221319 (+ 406us) server_negotiation.cc:407] Connection header received
0620 06:57:29.221457 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.221462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.221528 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.221608 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:29.222617 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.223380 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.224555 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.225036 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.225135 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.227108 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:29.227114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:29.227117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:29.227145 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:29.229870 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.230469 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.230475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.230479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.230677 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.231017 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.231022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.231026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.231310 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:29.231535 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.231825 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.232053 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 06:57:29.232913 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42254
W20260620 06:57:29.233450 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:29.233675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1582
I20260620 06:57:29.248996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1722
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:29.262843 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.249723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.0.97.2:42285)
0620 06:57:29.249902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.249906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.249923 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:57:29.250035 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.250056 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.250142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.250227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:29.251446 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.252196 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.253452 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.253836 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.253933 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.257650 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:29.257664 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:29.257680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:29.257713 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:29.260446 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.261042 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.261049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.261052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.261193 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.261440 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.261443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.261445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.261769 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:29.261980 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.262259 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.262432 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 06:57:29.263477 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47606
W20260620 06:57:29.264151 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:29.264274 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1722
I20260620 06:57:29.277297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1861
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938648, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:29.290634 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:29.277988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.0.97.3:44043)
0620 06:57:29.278142 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:29.278147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:29.278164 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:57:29.278313 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:29.278317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:29.278396 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:29.278478 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:29.279691 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.280422 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.281592 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:29.281997 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:29.282127 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:29.285844 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:29.285858 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:29.285874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:29.285912 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:29.288489 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.289062 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.289068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.289071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.289196 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:29.289449 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:29.289452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:29.289453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:29.289779 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:29.289970 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:29.290215 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 06:57:29.290372 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 06:57:29.291239 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43160
W20260620 06:57:29.291815 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:29.291980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1861
I20260620 06:57:29.308298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 1479
I20260620 06:57:29.309393 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:29.419045 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 1479
2026-06-20T06:57:29Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4405 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[2096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[2096](info): set up 2 sockets
Jun 20 06:57:29 dist-test-slave-mz20 krb5kdc[2096](info): commencing operation
krb5kdc: starting...
W20260620 06:57:30.426338 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:30 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:30.597946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39027
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:30.902490 2109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:30.902786 2109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:30.902990 2109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:30.910146 2109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:30.910250 2109 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:57:30.910291 2109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:30.910377 2109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:30.910440 2109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:30.910506 2109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:30.920147 2109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:30.921375 2109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:30.922701 2109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:30.933759 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:30.933797 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:30.933869 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:30.934872 2109 server_base.cc:1034] running on GCE node
I20260620 06:57:30.935590 2109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:30.936903 2109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:30.938117 2109 hybrid_clock.cc:648] HybridClock initialized: now 1781938650938057 us; error 34 us; skew 500 ppm
I20260620 06:57:30.938449 2109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:30 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:30.941335 2109 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:30.942687 2109 webserver.cc:460] Webserver started at http://127.0.97.62:34605/ using document root <none> and password file <none>
I20260620 06:57:30.943322 2109 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:30.943413 2109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:30.943689 2109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:30.946463 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "64e1c36822a64b0b8cafe9ed0d21cd03"
format_stamp: "Formatted at 2026-06-20 06:57:30 on dist-test-slave-mz20"
server_key: "cc2f1a401f97571e7f50bf1aecd63859"
server_key_iv: "884e94ab580f8790af3e3f66d8acc5cd"
server_key_version: "encryptionkey@0"
I20260620 06:57:30.947232 2109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "64e1c36822a64b0b8cafe9ed0d21cd03"
format_stamp: "Formatted at 2026-06-20 06:57:30 on dist-test-slave-mz20"
server_key: "cc2f1a401f97571e7f50bf1aecd63859"
server_key_iv: "884e94ab580f8790af3e3f66d8acc5cd"
server_key_version: "encryptionkey@0"
I20260620 06:57:30.952116 2109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:30.955361 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:30.956697 2109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:30.956876 2109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "64e1c36822a64b0b8cafe9ed0d21cd03"
format_stamp: "Formatted at 2026-06-20 06:57:30 on dist-test-slave-mz20"
server_key: "cc2f1a401f97571e7f50bf1aecd63859"
server_key_iv: "884e94ab580f8790af3e3f66d8acc5cd"
server_key_version: "encryptionkey@0"
I20260620 06:57:30.957079 2109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:30.973217 2109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:30.977355 2109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:30.977586 2109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:30.996654 2109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39027
I20260620 06:57:30.996675 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39027 every 8 connection(s)
I20260620 06:57:30.998016 2109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:31.001690 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:31.003222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2109
I20260620 06:57:31.003432 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:31.004026 388 external_mini_cluster.cc:1428] Setting key e605306a35bd7d34557a9530c6fc1273
I20260620 06:57:31.014317 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Bootstrap starting.
I20260620 06:57:31.017678 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Neither blocks nor log segments found. Creating new log.
Jun 20 06:57:31 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:31.018903 2186 log.cc:824] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:31.022306 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: No bootstrap required, opened a new log
I20260620 06:57:31.028112 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:31.006005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.0.97.62:39027)
0620 06:57:31.006890 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:31.006910 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:31.006974 (+ 64us) server_negotiation.cc:407] Connection header received
0620 06:57:31.009285 (+ 2311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:31.009315 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:31.009800 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:31.010242 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:31.011527 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.013491 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.014746 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.015710 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.015901 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:31.019833 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:31.019841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:31.019857 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:31.019891 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:31.023650 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.024337 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.024346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.024359 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.024560 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.024879 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.024884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.024887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.025618 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:31.025904 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:31.026222 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 06:57:31.026711 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":368,"threads_started":1}
I20260620 06:57:31.031625 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } }
I20260620 06:57:31.031942 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:31.032011 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e1c36822a64b0b8cafe9ed0d21cd03, State: Initialized, Role: FOLLOWER
I20260620 06:57:31.032482 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } }
I20260620 06:57:31.032672 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:31.032765 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:31.032881 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:31.035075 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } }
I20260620 06:57:31.035509 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64e1c36822a64b0b8cafe9ed0d21cd03; no voters:
I20260620 06:57:31.035983 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:31.036104 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:31.036725 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [term 1 LEADER]: Becoming Leader. State: Replica: 64e1c36822a64b0b8cafe9ed0d21cd03, State: Running, Role: LEADER
I20260620 06:57:31.037204 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } }
I20260620 06:57:31.037945 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:31.040117 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64e1c36822a64b0b8cafe9ed0d21cd03. Latest consensus state: current_term: 1 leader_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } } }
I20260620 06:57:31.040578 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:31.040143 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64e1c36822a64b0b8cafe9ed0d21cd03" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39027 } } }
I20260620 06:57:31.041021 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:31.041937 2199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:31.046481 2199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:31.054744 2199 catalog_manager.cc:1347] Generated new cluster ID: 07b85a31f04e4a4bbad1fc0fb51010e1
I20260620 06:57:31.054862 2199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:31.070127 2199 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:31.072017 2199 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:31.094126 2199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Generated new TSK 0
I20260620 06:57:31.095249 2199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:31.144407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39027
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:31.456672 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:31.456966 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:31.457116 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:31.464516 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:31.464637 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:31.464758 2212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:57:31.464818 2212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:31.464917 2212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:31.477353 2212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:31.478685 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:31.480005 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:31.492282 2218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:31.492355 2220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:31.492332 2217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:31.493206 2212 server_base.cc:1034] running on GCE node
I20260620 06:57:31.493793 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:31.494733 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:31.495922 2212 hybrid_clock.cc:648] HybridClock initialized: now 1781938651495866 us; error 54 us; skew 500 ppm
I20260620 06:57:31.496217 2212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:31 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:31.499166 2212 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:31.500622 2212 webserver.cc:460] Webserver started at http://127.0.97.1:44059/ using document root <none> and password file <none>
I20260620 06:57:31.501303 2212 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:31.501405 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:31.501655 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:31.504452 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "f71a44107ba951bba2201cbf00200b68"
server_key_iv: "7a2c485b3bc6e358b01f6b3e845cffc8"
server_key_version: "encryptionkey@0"
I20260620 06:57:31.505188 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "f71a44107ba951bba2201cbf00200b68"
server_key_iv: "7a2c485b3bc6e358b01f6b3e845cffc8"
server_key_version: "encryptionkey@0"
I20260620 06:57:31.510143 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:31.513585 2227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:31.514945 2212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:57:31.515087 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "f71a44107ba951bba2201cbf00200b68"
server_key_iv: "7a2c485b3bc6e358b01f6b3e845cffc8"
server_key_version: "encryptionkey@0"
I20260620 06:57:31.515268 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:31.546391 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:31.550740 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:31.550984 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:31.553469 2212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:31.553578 2212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:31.553686 2212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:31.553769 2212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:57:31 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:31.577581 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:31.554331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58151 (local address 127.0.97.62:39027)
0620 06:57:31.554783 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:31.554787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:31.557166 (+ 2379us) server_negotiation.cc:407] Connection header received
0620 06:57:31.559389 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:31.559396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:31.559495 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:31.559623 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:31.561871 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.562874 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.565367 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.565891 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.566002 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:31.570492 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:31.570497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:31.570500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:31.570530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:31.573726 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.574961 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.574967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.574971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.575143 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.575759 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.575765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.575769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.576126 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:31.576276 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:31.576990 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 06:57:31.577205 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":248,"threads_started":1}
I20260620 06:57:31.578946 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:31.555647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.1:58151)
0620 06:57:31.556576 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:31.556617 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:31.558891 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:31.559883 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:31.559891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:31.560797 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:31.561665 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:31.561692 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.563085 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.563089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:31.565232 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:31.565244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.566122 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.566129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:31.566362 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:31.567197 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:31.567236 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:31.570284 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:31.573927 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:31.573937 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:31.573960 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:31.574778 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:31.575279 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:31.575288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:31.575292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:31.575597 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:31.576264 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:31.576273 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:31.576624 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 06:57:31.576874 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 06:57:31.577579 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":374,"threads_started":1}
I20260620 06:57:31.595121 2212 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38245
I20260620 06:57:31.595152 2347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38245 every 8 connection(s)
I20260620 06:57:31.596410 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:31.598651 2348 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39027
I20260620 06:57:31.598892 2348 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:31.599543 2348 heartbeater.cc:511] Master 127.0.97.62:39027 requested a full tablet report, sending...
I20260620 06:57:31.601760 2141 ts_manager.cc:194] Registered new tserver with Master: ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245)
I20260620 06:57:31.602603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2212
I20260620 06:57:31.602793 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:31.603492 388 external_mini_cluster.cc:1428] Setting key dd306e3a51837b91880a36952a0a2142
I20260620 06:57:31.603682 2141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58151
I20260620 06:57:31.603861 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:31.590270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57191 (local address 127.0.97.62:39027)
0620 06:57:31.590480 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:31.590486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:31.591700 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 06:57:31.591914 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:31.591920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:31.592012 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:31.592129 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:31.593383 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.594114 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.595661 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.596061 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.596175 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:31.598230 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:31.598234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:31.598237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:31.598268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:31.601028 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.601649 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.601654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.601658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.601818 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:31.602222 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:31.602226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:31.602229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:31.602609 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:31.602746 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:31.603286 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 06:57:31.603507 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:57:31.603881 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:31.589952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.1:57191)
0620 06:57:31.591547 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:31.591573 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:31.591761 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:31.592304 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:31.592335 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:31.592934 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:31.593219 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:31.593229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.594305 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.594310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:31.595525 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:31.595539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:31.596246 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:31.596253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:31.596415 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:31.597137 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:31.597164 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:31.598101 (+ 937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:31.601181 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:31.601185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:31.601188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:31.601535 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:31.601985 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:31.601991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:31.601995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:31.602120 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:31.602806 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:31.602811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:31.603006 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 06:57:31.603148 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 06:57:31.603452 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1384,"thread_start_us":1105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:31.641722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39027
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:31.948544 2352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:31.948827 2352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:31.948997 2352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:31.956032 2352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:31.956146 2352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:31.956262 2352 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:57:31.956326 2352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:31.956429 2352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:31.965780 2352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:31.967104 2352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:31.968360 2352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:31.980388 2358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:31.980438 2360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:31.980515 2357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:31.981468 2352 server_base.cc:1034] running on GCE node
I20260620 06:57:31.982065 2352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:31.982963 2352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:31.984153 2352 hybrid_clock.cc:648] HybridClock initialized: now 1781938651984095 us; error 43 us; skew 500 ppm
I20260620 06:57:31.984436 2352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:31 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:31.987341 2352 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:31.988672 2352 webserver.cc:460] Webserver started at http://127.0.97.2:35007/ using document root <none> and password file <none>
I20260620 06:57:31.989315 2352 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:31.989427 2352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:31.989696 2352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:31.992416 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "0df52c0020284257bb59594c9e4423b9"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "ce7f46541db7313ce224fe11ad8b2f4e"
server_key_iv: "47e79fa787f1f412a340751e123b7221"
server_key_version: "encryptionkey@0"
I20260620 06:57:31.993141 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0df52c0020284257bb59594c9e4423b9"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "ce7f46541db7313ce224fe11ad8b2f4e"
server_key_iv: "47e79fa787f1f412a340751e123b7221"
server_key_version: "encryptionkey@0"
I20260620 06:57:31.998174 2352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:32.001813 2367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:32.003296 2352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:57:32.003436 2352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "0df52c0020284257bb59594c9e4423b9"
format_stamp: "Formatted at 2026-06-20 06:57:31 on dist-test-slave-mz20"
server_key: "ce7f46541db7313ce224fe11ad8b2f4e"
server_key_iv: "47e79fa787f1f412a340751e123b7221"
server_key_version: "encryptionkey@0"
I20260620 06:57:32.003600 2352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:32.029098 2352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:32.033171 2352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:32.033437 2352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:32.036110 2352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:32.036229 2352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:32.036358 2352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:32.036486 2352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 06:57:32 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:32.058769 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.036746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59121 (local address 127.0.97.62:39027)
0620 06:57:32.036931 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.036938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.038557 (+ 1619us) server_negotiation.cc:407] Connection header received
0620 06:57:32.040885 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.040892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.040973 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.041047 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:32.043396 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.044385 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.047113 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.047696 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.047817 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.052198 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:32.052203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:32.052206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:32.052235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:32.055467 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.056455 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.056460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.056462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.056601 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.057087 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.057092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.057096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.057360 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:32.057488 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.058181 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.058396 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:57:32.060139 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.037491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.2:59121)
0620 06:57:32.038336 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.038421 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.040517 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.041572 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.041584 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.042347 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:32.043224 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.043253 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.044584 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.044590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.046966 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.046981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.047955 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.047962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.048238 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.048996 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:32.049042 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:32.051966 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:32.055628 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.055638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.055658 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.056296 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.056747 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.056752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.056755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.056967 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.057488 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:32.057496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:32.057826 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.058054 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.058593 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":365,"threads_started":1}
I20260620 06:57:32.074822 2352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32963
I20260620 06:57:32.074908 2486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32963 every 8 connection(s)
I20260620 06:57:32.076143 2352 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:32.077381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2352
I20260620 06:57:32.077577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:32.078085 388 external_mini_cluster.cc:1428] Setting key e4556c7e379d1b16c80ed43b87a10564
I20260620 06:57:32.079283 2487 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39027
I20260620 06:57:32.079537 2487 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:32.081895 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.066614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36157 (local address 127.0.97.62:39027)
0620 06:57:32.066779 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.066784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.067469 (+ 685us) server_negotiation.cc:407] Connection header received
0620 06:57:32.067735 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.067740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.067816 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.067885 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:32.069411 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.070501 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.072676 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.073291 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.073372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.075898 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:32.075903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:32.075907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:32.076050 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:32.079317 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.080128 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.080132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.080134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.080244 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.080609 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.080612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.080614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.080969 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:32.081076 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.081512 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.081678 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:57:32.082135 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.066652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.2:36157)
0620 06:57:32.067363 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.067384 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.067584 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.068111 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.068115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.068764 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:32.069224 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.069241 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.070739 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.070744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.072484 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.072504 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.073692 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.073698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.073856 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.074745 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:32.074778 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:32.075687 (+ 909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:32.079474 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.079480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.079483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.080008 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.080348 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.080353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.080357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.080500 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.081085 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:32.081091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:32.081280 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.081425 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.081744 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":414,"threads_started":1}
I20260620 06:57:32.085386 2487 heartbeater.cc:511] Master 127.0.97.62:39027 requested a full tablet report, sending...
I20260620 06:57:32.086836 2140 ts_manager.cc:194] Registered new tserver with Master: 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963)
I20260620 06:57:32.087944 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59121
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:32.117391 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39027
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:32.425995 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:32.426256 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:32.426352 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:32.433450 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:32.433553 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:32.433645 2491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:57:32.433696 2491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:57:32.433763 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:32.443804 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:32.444989 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:32.446259 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:32.458474 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:32.458447 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:32.458683 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:32.459607 2491 server_base.cc:1034] running on GCE node
I20260620 06:57:32.460187 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:32.461100 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:32.462322 2491 hybrid_clock.cc:648] HybridClock initialized: now 1781938652462273 us; error 35 us; skew 500 ppm
I20260620 06:57:32.462646 2491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:32 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:32.465600 2491 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:32.466941 2491 webserver.cc:460] Webserver started at http://127.0.97.3:41003/ using document root <none> and password file <none>
I20260620 06:57:32.467635 2491 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:32.467743 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:32.468011 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:32.470806 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "fc304b0465f245049586d526b461692c"
format_stamp: "Formatted at 2026-06-20 06:57:32 on dist-test-slave-mz20"
server_key: "e71ceb14c708d38659a5cb95cb9fe2f0"
server_key_iv: "c09769930c02b5e3de03fc92f4266eec"
server_key_version: "encryptionkey@0"
I20260620 06:57:32.471517 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fc304b0465f245049586d526b461692c"
format_stamp: "Formatted at 2026-06-20 06:57:32 on dist-test-slave-mz20"
server_key: "e71ceb14c708d38659a5cb95cb9fe2f0"
server_key_iv: "c09769930c02b5e3de03fc92f4266eec"
server_key_version: "encryptionkey@0"
I20260620 06:57:32.476297 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:57:32.479540 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:32.480852 2491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:32.481005 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "fc304b0465f245049586d526b461692c"
format_stamp: "Formatted at 2026-06-20 06:57:32 on dist-test-slave-mz20"
server_key: "e71ceb14c708d38659a5cb95cb9fe2f0"
server_key_iv: "c09769930c02b5e3de03fc92f4266eec"
server_key_version: "encryptionkey@0"
I20260620 06:57:32.481175 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:32.497586 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:32.501236 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:32.501485 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:32.504015 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:32.504113 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:32.504227 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:32.504299 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:57:32 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:32.527500 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.504719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57363 (local address 127.0.97.62:39027)
0620 06:57:32.504915 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.504921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.506310 (+ 1389us) server_negotiation.cc:407] Connection header received
0620 06:57:32.508502 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.508509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.508597 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.508722 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:32.510917 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.511870 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.514231 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.514764 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.514889 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.519549 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:32.519556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:32.519560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:32.519603 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:32.522571 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.523683 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.523688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.523692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.523863 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.524420 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.524426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.524429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.524715 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:32.524857 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.526767 (+ 1910us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.527011 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:57:32.527654 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.505266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.3:57363)
0620 06:57:32.506118 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.506174 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.508164 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.509020 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.509033 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.509808 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:32.510719 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.510746 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.512093 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.512099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.514059 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.514071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.515151 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.515159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.515403 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.516164 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:32.516186 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:32.519254 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:32.522760 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.522774 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.522794 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.523517 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.523965 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.523974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.523978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.524258 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.524935 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:32.524944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:32.525461 (+ 517us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.525792 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.526304 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":372,"threads_started":1}
I20260620 06:57:32.547748 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.534261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36133 (local address 127.0.97.62:39027)
0620 06:57:32.534448 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.534455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.535021 (+ 566us) server_negotiation.cc:407] Connection header received
0620 06:57:32.535179 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.535185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.535266 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.535378 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:32.536616 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.537698 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.539707 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.540246 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.540364 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.542806 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:32.542811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:32.542815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:32.542844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:32.545151 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.545783 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.545789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.545792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.545933 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.546267 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.546272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.546276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.546594 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:32.546730 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.547203 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.547407 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:57:32.547857 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.534252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.97.3:36133)
0620 06:57:32.534924 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.534943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.535067 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.535426 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.535430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.536083 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:32.536489 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.536503 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.537850 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.537859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.539556 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.539572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.540650 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.540661 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.540792 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.541508 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:32.541528 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:32.542596 (+ 1068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:32.545272 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.545277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.545282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.545667 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.546024 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:32.546028 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:32.546032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:32.546163 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:32.546707 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:32.546712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:32.546911 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.547074 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.547386 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":373,"threads_started":1}
I20260620 06:57:32.550942 2491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42449
I20260620 06:57:32.550959 2625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42449 every 8 connection(s)
I20260620 06:57:32.552141 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:32.553985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2491
I20260620 06:57:32.554124 2626 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39027
I20260620 06:57:32.554188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:32.554347 2626 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:32.554702 388 external_mini_cluster.cc:1428] Setting key cd36c13eed22f9ac738fe1bfe1b5c8da
I20260620 06:57:32.555124 2626 heartbeater.cc:511] Master 127.0.97.62:39027 requested a full tablet report, sending...
I20260620 06:57:32.556428 2140 ts_manager.cc:194] Registered new tserver with Master: fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:32.557276 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57363
I20260620 06:57:32.557574 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:57:32 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:57:32 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:32.599483 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.586279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.0.97.62:39027)
0620 06:57:32.586489 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.586495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.586662 (+ 167us) server_negotiation.cc:407] Connection header received
0620 06:57:32.586833 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.586837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.586917 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.586995 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:32.588322 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.589071 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.590262 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.590704 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.590793 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.594450 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:32.594455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:32.594459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:32.594493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:32.596938 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.597648 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.597652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.597654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.597762 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:32.598106 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:32.598109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:32.598111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:32.598343 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:32.598618 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.598918 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.599132 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:57:32.607144 2348 heartbeater.cc:503] Master 127.0.97.62:39027 was elected leader, sending a full tablet report...
I20260620 06:57:32.603183 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:32.610076 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:32.641673 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.621021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.0.97.1:38245)
0620 06:57:32.633874 (+ 12853us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.633881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.633904 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:57:32.633975 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.633981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.634181 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.634325 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:32.635196 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.636583 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.639251 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.640573 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.640695 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.640909 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.641222 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.641380 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12720,"thread_start_us":12601,"threads_started":1}
I20260620 06:57:32.641707 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.620821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38245 (local address 127.0.0.1:53704)
0620 06:57:32.632826 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.632886 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.633086 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.634388 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.634397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.634433 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:32.634966 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.634980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.636702 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.636707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.639075 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.639089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.640771 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.640777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.641010 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.641047 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.641154 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.641337 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11719,"mutex_wait_us":23,"thread_start_us":13811,"threads_started":1}
I20260620 06:57:32.642853 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.620992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32963 (local address 127.0.0.1:52166)
0620 06:57:32.634861 (+ 13869us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.634875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.634988 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.635751 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.635756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.635780 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:32.636205 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.636220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.637907 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.637912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.640110 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.640123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.642049 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.642055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.642212 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.642244 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.642334 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.642462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13773,"mutex_wait_us":3,"thread_start_us":11767,"threads_started":1}
I20260620 06:57:32.642879 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.621444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.0.97.2:32963)
0620 06:57:32.632227 (+ 10783us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.632239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.634973 (+ 2734us) server_negotiation.cc:407] Connection header received
0620 06:57:32.635134 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.635141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.635372 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.635549 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:32.636444 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.637781 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.640279 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.641620 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.641729 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.641914 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.642380 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.642594 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10567,"thread_start_us":11842,"threads_started":1}
I20260620 06:57:32.644148 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.635309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.0.97.3:42449)
0620 06:57:32.635885 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.635893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.636565 (+ 672us) server_negotiation.cc:407] Connection header received
0620 06:57:32.637024 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.637032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.637235 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.637377 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:32.638311 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.639401 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.641551 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.642997 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.643088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.643295 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.643798 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.643916 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
I20260620 06:57:32.644181 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.635880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.0.1:58840)
0620 06:57:32.636443 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.636475 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.636838 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.637650 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.637655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.637677 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:32.638134 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.638144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.639532 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.639537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.641371 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.641385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.643474 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.643481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.643639 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.643667 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.643757 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.643867 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":27,"thread_start_us":323,"threads_started":1}
I20260620 06:57:32.647277 2417 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7349268394872987f0340b7a8f755 (DEFAULT_TABLE table=test-table [id=c76e7e8c6c7045738bc21124531450ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:32.647284 2278 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7349268394872987f0340b7a8f755 (DEFAULT_TABLE table=test-table [id=c76e7e8c6c7045738bc21124531450ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:32.647677 2556 tablet_service.cc:1467] Processing CreateTablet for tablet bcd7349268394872987f0340b7a8f755 (DEFAULT_TABLE table=test-table [id=c76e7e8c6c7045738bc21124531450ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:32.648456 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7349268394872987f0340b7a8f755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:32.648588 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7349268394872987f0340b7a8f755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:32.648859 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd7349268394872987f0340b7a8f755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:32.660574 2645 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Bootstrap starting.
I20260620 06:57:32.662787 2647 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Bootstrap starting.
I20260620 06:57:32.664206 2645 tablet_bootstrap.cc:654] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:32.665524 2645 log.cc:824] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:32.666265 2647 tablet_bootstrap.cc:654] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:32.667536 2647 log.cc:824] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:32.668485 2645 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: No bootstrap required, opened a new log
I20260620 06:57:32.668756 2645 ts_tablet_manager.cc:1397] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 06:57:32.670179 2647 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: No bootstrap required, opened a new log
I20260620 06:57:32.670418 2647 ts_tablet_manager.cc:1397] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 06:57:32.670646 2646 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Bootstrap starting.
I20260620 06:57:32.673237 2646 tablet_bootstrap.cc:654] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:32.674162 2646 log.cc:824] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:32.676424 2646 tablet_bootstrap.cc:492] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: No bootstrap required, opened a new log
I20260620 06:57:32.676651 2646 ts_tablet_manager.cc:1397] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260620 06:57:32.678776 2645 raft_consensus.cc:348] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.679206 2645 raft_consensus.cc:374] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:32.679301 2645 raft_consensus.cc:729] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Initialized, Role: FOLLOWER
I20260620 06:57:32.679883 2645 consensus_queue.cc:260] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.680308 2647 raft_consensus.cc:348] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.680766 2647 raft_consensus.cc:374] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:32.680930 2647 raft_consensus.cc:729] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Initialized, Role: FOLLOWER
I20260620 06:57:32.680941 2487 heartbeater.cc:503] Master 127.0.97.62:39027 was elected leader, sending a full tablet report...
I20260620 06:57:32.681622 2645 ts_tablet_manager.cc:1428] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260620 06:57:32.681604 2647 consensus_queue.cc:260] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.684906 2646 raft_consensus.cc:348] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.685352 2646 raft_consensus.cc:374] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:32.685485 2646 raft_consensus.cc:729] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc304b0465f245049586d526b461692c, State: Initialized, Role: FOLLOWER
I20260620 06:57:32.686077 2646 consensus_queue.cc:260] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.687711 2647 ts_tablet_manager.cc:1428] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260620 06:57:32.688715 2626 heartbeater.cc:503] Master 127.0.97.62:39027 was elected leader, sending a full tablet report...
I20260620 06:57:32.689440 2646 ts_tablet_manager.cc:1428] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
W20260620 06:57:32.803778 2627 tablet.cc:2369] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:57:32.828882 2488 tablet.cc:2369] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:57:32.848742 2349 tablet.cc:2369] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:57:32.946938 2651 raft_consensus.cc:482] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:32.947239 2651 raft_consensus.cc:504] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.948810 2651 leader_election.cc:288] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:32.954713 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.949164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49445 (local address 127.0.97.1:38245)
0620 06:57:32.949389 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.949396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.949717 (+ 321us) server_negotiation.cc:407] Connection header received
0620 06:57:32.949924 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.949930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.950022 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.950110 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:32.950800 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.951494 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.953151 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.953879 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.953992 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.954099 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.954355 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.954471 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:57:32.954805 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.949062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38245 (local address 127.0.97.2:49445)
0620 06:57:32.949598 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.949613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.949786 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.950179 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.950185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.950214 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:32.950644 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.950655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.951621 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.951627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.952996 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.953008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.954074 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.954080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.954215 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.954238 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.954316 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.954441 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":244,"threads_started":1}
I20260620 06:57:32.955088 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.949894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58745 (local address 127.0.97.3:42449)
0620 06:57:32.950092 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:32.950097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:32.950415 (+ 318us) server_negotiation.cc:407] Connection header received
0620 06:57:32.950586 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:32.950591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:32.950654 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:32.950728 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:32.951207 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.951875 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.953484 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.954236 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.954344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.954441 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:32.954742 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 06:57:32.954861 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 06:57:32.955200 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:32.949689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.97.2:58745)
0620 06:57:32.950304 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:32.950324 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:32.950438 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:32.950756 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:32.950760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:32.950782 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:32.951064 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.951072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.952007 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.952012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.953346 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:32.953356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:32.954427 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:32.954434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:32.954601 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:32.954630 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:57:32.954715 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:57:32.954839 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":3,"thread_start_us":304,"threads_started":1}
I20260620 06:57:32.955960 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7349268394872987f0340b7a8f755" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" is_pre_election: true
I20260620 06:57:32.956184 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7349268394872987f0340b7a8f755" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c" is_pre_election: true
I20260620 06:57:32.956478 2298 raft_consensus.cc:2454] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 0.
I20260620 06:57:32.956662 2576 raft_consensus.cc:2454] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 0.
I20260620 06:57:32.957078 2371 leader_election.cc:302] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, ce8abf9305214ecdb778167a1a4cd3ad; no voters:
I20260620 06:57:32.957404 2651 raft_consensus.cc:2781] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:32.957523 2651 raft_consensus.cc:482] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:32.957629 2651 raft_consensus.cc:3037] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:32.959717 2651 raft_consensus.cc:504] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.960242 2651 leader_election.cc:288] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 election: Requested vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:32.960637 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7349268394872987f0340b7a8f755" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
I20260620 06:57:32.960686 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd7349268394872987f0340b7a8f755" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c"
I20260620 06:57:32.960813 2298 raft_consensus.cc:3037] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:32.960866 2576 raft_consensus.cc:3037] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:32.963192 2576 raft_consensus.cc:2454] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 1.
I20260620 06:57:32.963207 2298 raft_consensus.cc:2454] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 1.
I20260620 06:57:32.963572 2371 leader_election.cc:302] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:32.963847 2651 raft_consensus.cc:2781] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:32.964251 2651 raft_consensus.cc:686] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Running, Role: LEADER
I20260620 06:57:32.964747 2651 consensus_queue.cc:237] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:32.968739 2140 catalog_manager.cc:5495] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0df52c0020284257bb59594c9e4423b9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0df52c0020284257bb59594c9e4423b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:57:33 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:33.025346 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.012942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.0.97.62:39027)
0620 06:57:33.013126 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.013131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.013645 (+ 514us) server_negotiation.cc:407] Connection header received
0620 06:57:33.013799 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.013804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.013868 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.013944 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:33.015126 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.015840 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.016984 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.017376 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.017475 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.020676 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:33.020681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:33.020686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:33.020717 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:33.023097 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:33.023728 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:33.023731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:33.023734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:33.023839 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:33.024154 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:33.024157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:33.024159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:33.024415 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:33.024516 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.024867 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.025074 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":63}
I20260620 06:57:33.025377 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.012920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39027 (local address 127.0.0.1:33320)
0620 06:57:33.013524 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.013562 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.013696 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.013988 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.013992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.014735 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:33.014998 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.015008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.015957 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.015960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.016864 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.016874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.017553 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.017559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.017696 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.018332 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:33.018350 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:33.020499 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:33.023223 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:33.023232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:33.023238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:33.023634 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:33.023927 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:33.023931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:33.023933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:33.024075 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:33.024509 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:33.024515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:33.024660 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.024785 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.024974 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":351,"threads_started":1}
I20260620 06:57:33.030218 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:57:33.031271 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:33.037739 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2547dc35f64a418aaa077a1badb2fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01289adb167448fb84838b454379d0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:33.038662 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2547dc35f64a418aaa077a1badb2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.037671 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2547dc35f64a418aaa077a1badb2fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01289adb167448fb84838b454379d0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:33.038100 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 1c2547dc35f64a418aaa077a1badb2fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e01289adb167448fb84838b454379d0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:57:33.039113 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2547dc35f64a418aaa077a1badb2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.039112 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c2547dc35f64a418aaa077a1badb2fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.043783 2646 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: Bootstrap starting.
I20260620 06:57:33.045331 2646 tablet_bootstrap.cc:654] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.045919 2645 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: Bootstrap starting.
I20260620 06:57:33.046518 2647 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: Bootstrap starting.
I20260620 06:57:33.047281 2646 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: No bootstrap required, opened a new log
I20260620 06:57:33.047384 2646 ts_tablet_manager.cc:1397] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 06:57:33.047518 2645 tablet_bootstrap.cc:654] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.048022 2647 tablet_bootstrap.cc:654] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.047982 2646 raft_consensus.cc:348] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.048169 2646 raft_consensus.cc:374] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.048235 2646 raft_consensus.cc:729] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc304b0465f245049586d526b461692c, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.048492 2646 consensus_queue.cc:260] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.048956 2646 ts_tablet_manager.cc:1428] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 06:57:33.048861 2653 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.049144 2653 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.050452 2647 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: No bootstrap required, opened a new log
I20260620 06:57:33.050611 2647 ts_tablet_manager.cc:1397] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260620 06:57:33.051396 2645 tablet_bootstrap.cc:492] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: No bootstrap required, opened a new log
I20260620 06:57:33.051331 2647 raft_consensus.cc:348] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.051532 2645 ts_tablet_manager.cc:1397] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:57:33.051566 2647 raft_consensus.cc:374] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.051656 2647 raft_consensus.cc:729] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.051879 2647 consensus_queue.cc:260] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.052238 2645 raft_consensus.cc:348] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.052465 2647 ts_tablet_manager.cc:1428] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 06:57:33.052517 2645 raft_consensus.cc:374] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.052613 2645 raft_consensus.cc:729] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.052359 2652 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.052747 2652 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.052908 2645 consensus_queue.cc:260] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.053630 2651 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.053918 2651 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.054661 2645 ts_tablet_manager.cc:1428] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 06:57:33.099960 2653 raft_consensus.cc:482] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:33.100143 2653 raft_consensus.cc:504] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.101667 2653 leader_election.cc:288] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963)
I20260620 06:57:33.107285 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.101933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39909 (local address 127.0.97.1:38245)
0620 06:57:33.102115 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.102121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.102570 (+ 449us) server_negotiation.cc:407] Connection header received
0620 06:57:33.102771 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.102777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.102868 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.102951 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:33.103336 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.104056 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.105651 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.106403 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.106510 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.106633 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.106880 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.107016 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:57:33.107366 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.101828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38245 (local address 127.0.97.3:39909)
0620 06:57:33.102445 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.102461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.102632 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.102978 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.102982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.103008 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:33.103190 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.103198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.104195 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.104200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.105508 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.105519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.106572 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.106579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.106745 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.106768 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.106846 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.106976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":3,"thread_start_us":695,"threads_started":1}
I20260620 06:57:33.107710 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.101930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32963 (local address 127.0.97.3:51003)
0620 06:57:33.102801 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.102818 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.102934 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.103216 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.103221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.103243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:33.103511 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.103519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.104463 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.104469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.105769 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.105780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.106956 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.106962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.107163 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.107187 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.107275 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.107407 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":291,"threads_started":1}
I20260620 06:57:33.107674 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.102313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51003 (local address 127.0.97.2:32963)
0620 06:57:33.102488 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.102494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.102905 (+ 411us) server_negotiation.cc:407] Connection header received
0620 06:57:33.103048 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.103051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.103107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.103177 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:33.103643 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.104338 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.105912 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.106761 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.106874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.106983 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.107310 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.107429 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:57:33.108263 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2547dc35f64a418aaa077a1badb2fb" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" is_pre_election: true
I20260620 06:57:33.108489 2298 raft_consensus.cc:2454] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 0.
I20260620 06:57:33.108713 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2547dc35f64a418aaa077a1badb2fb" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9" is_pre_election: true
I20260620 06:57:33.108933 2510 leader_election.cc:302] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce8abf9305214ecdb778167a1a4cd3ad, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:33.109154 2437 raft_consensus.cc:2454] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 0.
I20260620 06:57:33.109373 2653 raft_consensus.cc:2781] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:33.109490 2653 raft_consensus.cc:482] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:33.109601 2653 raft_consensus.cc:3037] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.111917 2653 raft_consensus.cc:504] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.112416 2653 leader_election.cc:288] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 election: Requested vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963)
I20260620 06:57:33.112876 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2547dc35f64a418aaa077a1badb2fb" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
I20260620 06:57:33.112892 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c2547dc35f64a418aaa077a1badb2fb" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9"
I20260620 06:57:33.113075 2437 raft_consensus.cc:3037] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.113094 2298 raft_consensus.cc:3037] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.115952 2437 raft_consensus.cc:2454] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 1.
I20260620 06:57:33.115939 2298 raft_consensus.cc:2454] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 1.
I20260620 06:57:33.116336 2508 leader_election.cc:302] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:33.116638 2653 raft_consensus.cc:2781] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:33.117031 2653 raft_consensus.cc:686] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [term 1 LEADER]: Becoming Leader. State: Replica: fc304b0465f245049586d526b461692c, State: Running, Role: LEADER
I20260620 06:57:33.117496 2653 consensus_queue.cc:237] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.120289 2669 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: TxnStatusTablet state changed. Reason: New leader fc304b0465f245049586d526b461692c. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.120594 2669 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:33.121348 2671 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:57:33.121588 2671 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:57:33.122206 2139 catalog_manager.cc:5495] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c reported cstate change: term changed from 0 to 1, leader changed from <none> to fc304b0465f245049586d526b461692c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } health_report { overall_health: HEALTHY } } }
I20260620 06:57:33.142668 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.136448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.0.1:58844)
0620 06:57:33.136701 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.136721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.136871 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.137342 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.137345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.137355 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:33.137511 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.137518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.138979 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.138983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.140030 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.140042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.140947 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.140951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.141104 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.141159 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.142196 (+ 1037us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.142234 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.142298 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.142415 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:57:33.142758 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.136691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.0.97.3:42449)
0620 06:57:33.136894 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.136900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.136924 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:33.137056 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.137062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.137171 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.137333 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.137650 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.138858 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.140164 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.140811 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.140939 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.141341 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.142090 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.142226 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.142329 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.142415 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:57:33.150430 2298 raft_consensus.cc:1261] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Refusing update from remote peer fc304b0465f245049586d526b461692c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.150519 2437 raft_consensus.cc:1261] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Refusing update from remote peer fc304b0465f245049586d526b461692c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.151417 2653 consensus_queue.cc:1035] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.151747 2669 consensus_queue.cc:1035] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.159190 2651 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: TxnStatusTablet state changed. Reason: New leader fc304b0465f245049586d526b461692c. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.159567 2651 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.163776 2674 mvcc.cc:205] Tried to move back new op lower bound from 7298820723294752768 to 7298820723172839424. Current Snapshot: MvccSnapshot[applied={T|T < 7298820723294752768}]
I20260620 06:57:33.164647 2652 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: TxnStatusTablet state changed. Reason: New leader fc304b0465f245049586d526b461692c. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.164934 2652 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.165926 2675 mvcc.cc:205] Tried to move back new op lower bound from 7298820723294752768 to 7298820723172839424. Current Snapshot: MvccSnapshot[applied={T|T < 7298820723294752768}]
I20260620 06:57:33.166532 2653 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.166824 2653 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:33.167063 2653 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.167279 2653 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P fc304b0465f245049586d526b461692c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:57:33.171053 2651 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.171312 2651 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P 0df52c0020284257bb59594c9e4423b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.179039 2652 tablet_replica.cc:439] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } } }
I20260620 06:57:33.179328 2652 tablet_replica.cc:442] T 1c2547dc35f64a418aaa077a1badb2fb P ce8abf9305214ecdb778167a1a4cd3ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:57:33.183027 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.175641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.97.2:32963)
0620 06:57:33.176074 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.176081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.176352 (+ 271us) server_negotiation.cc:407] Connection header received
0620 06:57:33.176559 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.176565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.176668 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.176828 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.177280 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.178374 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.180253 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.180806 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.180908 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.181740 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.182281 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.182362 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.182598 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.182688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260620 06:57:33.198223 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.193139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51705 (local address 127.0.97.3:42449)
0620 06:57:33.193293 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.193297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.193768 (+ 471us) server_negotiation.cc:407] Connection header received
0620 06:57:33.193954 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.193957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.194037 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.194109 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.194595 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.195283 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.196592 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.197004 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.197107 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.197524 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.197630 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.197703 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.197828 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.197903 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:57:33.198288 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.193059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.97.2:51705)
0620 06:57:33.193658 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.193672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.193822 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.194134 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.194138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.194149 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:33.194424 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.194432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.195410 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.195415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.196463 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.196473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.197165 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.197171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.197287 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.197336 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.197727 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.197737 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.197801 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.197944 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":372,"threads_started":1}
I20260620 06:57:33.208323 2298 raft_consensus.cc:1261] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Refusing update from remote peer 0df52c0020284257bb59594c9e4423b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.208566 2576 raft_consensus.cc:1261] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Refusing update from remote peer 0df52c0020284257bb59594c9e4423b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.208894 2656 consensus_queue.cc:1035] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.209285 2651 consensus_queue.cc:1035] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.252879 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.240342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32963 (local address 127.0.97.3:47553)
0620 06:57:33.246795 (+ 6453us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.246813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.246958 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.247556 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.247560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.247571 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:33.247966 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.247978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.249399 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.249403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.250791 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.250807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.251708 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.251715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.251871 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.251945 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.252404 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.252413 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.252491 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.252600 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6242,"thread_start_us":6104,"threads_started":1}
I20260620 06:57:33.252892 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.240342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47553 (local address 127.0.97.2:32963)
0620 06:57:33.240510 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.240517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.246885 (+ 6368us) server_negotiation.cc:407] Connection header received
0620 06:57:33.247096 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.247101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.247194 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.247289 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.248137 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.249276 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.250929 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.251540 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.251663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.252151 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.252275 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.252328 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.252537 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.252617 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:57:33.287178 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:33.287473 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:33.291378 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Sending DeleteTablet for 3 replicas of tablet bcd7349268394872987f0340b7a8f755
I20260620 06:57:33.292539 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7349268394872987f0340b7a8f755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53704
I20260620 06:57:33.293275 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7349268394872987f0340b7a8f755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52166
I20260620 06:57:33.293447 2696 tablet_replica.cc:330] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: stopping tablet replica
I20260620 06:57:33.293998 2696 raft_consensus.cc:2229] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:33.294498 2696 raft_consensus.cc:2258] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:33.296562 2696 ts_tablet_manager.cc:1905] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:33.297571 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd7349268394872987f0340b7a8f755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:33 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260620 06:57:33.304304 2696 ts_tablet_manager.cc:1918] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:33.304523 2696 log.cc:1196] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/bcd7349268394872987f0340b7a8f755
I20260620 06:57:33.305121 2696 ts_tablet_manager.cc:1939] T bcd7349268394872987f0340b7a8f755 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting consensus metadata
I20260620 06:57:33.307103 2128 catalog_manager.cc:4841] TS ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245): tablet bcd7349268394872987f0340b7a8f755 (table test-table [id=c76e7e8c6c7045738bc21124531450ba]) successfully deleted
I20260620 06:57:33.307070 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:33.307756 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:33.316612 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 5d861bfac35f49578910db210f17343b (DEFAULT_TABLE table=test-table [id=02ea60337e334ed0bb43c981dcb45f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:33.317242 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d861bfac35f49578910db210f17343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.317153 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 5d861bfac35f49578910db210f17343b (DEFAULT_TABLE table=test-table [id=02ea60337e334ed0bb43c981dcb45f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:33.317224 2555 tablet_service.cc:1467] Processing CreateTablet for tablet 5d861bfac35f49578910db210f17343b (DEFAULT_TABLE table=test-table [id=02ea60337e334ed0bb43c981dcb45f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:33.317586 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d861bfac35f49578910db210f17343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.317742 2555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d861bfac35f49578910db210f17343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:33.324168 2647 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Bootstrap starting.
I20260620 06:57:33.324302 2645 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Bootstrap starting.
I20260620 06:57:33.325438 2646 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Bootstrap starting.
I20260620 06:57:33.325768 2647 tablet_bootstrap.cc:654] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.325870 2645 tablet_bootstrap.cc:654] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.327005 2646 tablet_bootstrap.cc:654] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:33.335899 2647 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: No bootstrap required, opened a new log
I20260620 06:57:33.336145 2647 ts_tablet_manager.cc:1397] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260620 06:57:33.336910 2647 raft_consensus.cc:348] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.337205 2647 raft_consensus.cc:374] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.337312 2647 raft_consensus.cc:729] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.337849 2647 consensus_queue.cc:260] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.341213 2647 ts_tablet_manager.cc:1428] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 06:57:33.345098 2646 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: No bootstrap required, opened a new log
I20260620 06:57:33.345286 2646 ts_tablet_manager.cc:1397] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20260620 06:57:33.346095 2646 raft_consensus.cc:348] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.346352 2646 raft_consensus.cc:374] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.346431 2646 raft_consensus.cc:729] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc304b0465f245049586d526b461692c, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.346686 2646 consensus_queue.cc:260] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.348464 2646 ts_tablet_manager.cc:1428] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 06:57:33.350584 2698 tablet_replica.cc:330] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: stopping tablet replica
I20260620 06:57:33.351063 2698 raft_consensus.cc:2229] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:33.351647 2697 tablet_replica.cc:330] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: stopping tablet replica
I20260620 06:57:33.352069 2697 raft_consensus.cc:2229] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:33.352737 2697 raft_consensus.cc:2258] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:33.353094 2698 raft_consensus.cc:2258] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:33.355010 2698 ts_tablet_manager.cc:1905] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:33.359926 2645 tablet_bootstrap.cc:492] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: No bootstrap required, opened a new log
I20260620 06:57:33.360163 2645 ts_tablet_manager.cc:1397] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.005s
I20260620 06:57:33.361037 2697 ts_tablet_manager.cc:1905] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:33.361004 2645 raft_consensus.cc:348] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.361374 2645 raft_consensus.cc:374] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:33.361495 2645 raft_consensus.cc:729] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Initialized, Role: FOLLOWER
I20260620 06:57:33.361845 2645 consensus_queue.cc:260] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.362517 2645 ts_tablet_manager.cc:1428] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:57:33.366940 2698 ts_tablet_manager.cc:1918] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:33.367175 2698 log.cc:1196] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/bcd7349268394872987f0340b7a8f755
I20260620 06:57:33.367798 2698 ts_tablet_manager.cc:1939] T bcd7349268394872987f0340b7a8f755 P fc304b0465f245049586d526b461692c: Deleting consensus metadata
I20260620 06:57:33.369481 2128 catalog_manager.cc:4841] TS fc304b0465f245049586d526b461692c (127.0.97.3:42449): tablet bcd7349268394872987f0340b7a8f755 (table test-table [id=c76e7e8c6c7045738bc21124531450ba]) successfully deleted
I20260620 06:57:33.370688 2697 ts_tablet_manager.cc:1918] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:33.370859 2697 log.cc:1196] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/bcd7349268394872987f0340b7a8f755
I20260620 06:57:33.371351 2697 ts_tablet_manager.cc:1939] T bcd7349268394872987f0340b7a8f755 P 0df52c0020284257bb59594c9e4423b9: Deleting consensus metadata
I20260620 06:57:33.372952 2126 catalog_manager.cc:4841] TS 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963): tablet bcd7349268394872987f0340b7a8f755 (table test-table [id=c76e7e8c6c7045738bc21124531450ba]) successfully deleted
I20260620 06:57:33.425232 2653 raft_consensus.cc:482] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:33.425452 2653 raft_consensus.cc:504] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.426198 2653 leader_election.cc:288] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963)
I20260620 06:57:33.426841 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d861bfac35f49578910db210f17343b" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" is_pre_election: true
I20260620 06:57:33.427016 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d861bfac35f49578910db210f17343b" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9" is_pre_election: true
I20260620 06:57:33.427084 2298 raft_consensus.cc:2454] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 0.
I20260620 06:57:33.427238 2437 raft_consensus.cc:2454] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 0.
I20260620 06:57:33.427567 2510 leader_election.cc:302] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce8abf9305214ecdb778167a1a4cd3ad, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:33.427852 2653 raft_consensus.cc:2781] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:33.427963 2653 raft_consensus.cc:482] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:33.428048 2653 raft_consensus.cc:3037] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.431223 2653 raft_consensus.cc:504] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.432214 2653 leader_election.cc:288] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 election: Requested vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963)
I20260620 06:57:33.432330 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d861bfac35f49578910db210f17343b" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9"
I20260620 06:57:33.432531 2437 raft_consensus.cc:3037] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.432606 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d861bfac35f49578910db210f17343b" candidate_uuid: "fc304b0465f245049586d526b461692c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
I20260620 06:57:33.432796 2298 raft_consensus.cc:3037] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:33.435514 2437 raft_consensus.cc:2454] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 1.
I20260620 06:57:33.435570 2298 raft_consensus.cc:2454] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc304b0465f245049586d526b461692c in term 1.
I20260620 06:57:33.435972 2508 leader_election.cc:302] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:33.436285 2653 raft_consensus.cc:2781] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:33.436798 2653 raft_consensus.cc:686] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 1 LEADER]: Becoming Leader. State: Replica: fc304b0465f245049586d526b461692c, State: Running, Role: LEADER
I20260620 06:57:33.437083 2653 consensus_queue.cc:237] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:33.440032 2141 catalog_manager.cc:5495] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c reported cstate change: term changed from 0 to 1, leader changed from <none> to fc304b0465f245049586d526b461692c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "fc304b0465f245049586d526b461692c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } health_report { overall_health: HEALTHY } } }
I20260620 06:57:33.477821 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.470820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.0.97.3:42449)
0620 06:57:33.471388 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.471394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.471414 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:57:33.471486 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.471491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.471576 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.471682 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.472245 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.473340 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.475275 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.475998 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.476121 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.476705 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.476872 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.476956 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.477415 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.477497 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260620 06:57:33.497480 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.491357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53193 (local address 127.0.97.3:42449)
0620 06:57:33.491531 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:33.491536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:33.491588 (+ 52us) server_negotiation.cc:407] Connection header received
0620 06:57:33.491791 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:33.491797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:33.491879 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:33.491972 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:33.492356 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.493507 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.495462 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.496132 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.496247 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.496715 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.496844 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.496923 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:33.497082 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 06:57:33.497155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:57:33.497526 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:33.491285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.97.3:53193)
0620 06:57:33.491518 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:33.491536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:33.491660 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:33.492011 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:33.492016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:33.492029 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:33.492215 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.492225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.493696 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.493702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.495319 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:33.495338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:33.496285 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:33.496292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:33.496481 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:33.496530 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:33.496954 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:33.496966 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 06:57:33.497054 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:57:33.497189 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 06:57:33.507890 2437 raft_consensus.cc:1261] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Refusing update from remote peer fc304b0465f245049586d526b461692c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.509095 2298 raft_consensus.cc:1261] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Refusing update from remote peer fc304b0465f245049586d526b461692c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:33.509689 2653 consensus_queue.cc:1035] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.510176 2669 consensus_queue.cc:1035] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:33.527839 2676 mvcc.cc:205] Tried to move back new op lower bound from 7298820724760653824 to 7298820724480450560. Current Snapshot: MvccSnapshot[applied={T|T < 7298820724760653824}]
W20260620 06:57:33.554888 2674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:57:35.164927 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:35.165151 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:35.168658 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Sending DeleteTablet for 3 replicas of tablet 5d861bfac35f49578910db210f17343b
I20260620 06:57:35.169317 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d861bfac35f49578910db210f17343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53704
I20260620 06:57:35.169396 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d861bfac35f49578910db210f17343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260620 06:57:35.169405 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d861bfac35f49578910db210f17343b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52166
I20260620 06:57:35.179703 2728 tablet_replica.cc:330] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: stopping tablet replica
I20260620 06:57:35.179944 2728 raft_consensus.cc:2229] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:35.180203 2728 raft_consensus.cc:2258] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.180701 2727 tablet_replica.cc:330] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: stopping tablet replica
I20260620 06:57:35.180869 2727 raft_consensus.cc:2229] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:35.181074 2727 raft_consensus.cc:2258] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.181238 2728 ts_tablet_manager.cc:1905] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.181607 2729 tablet_replica.cc:330] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: stopping tablet replica
I20260620 06:57:35.181818 2729 raft_consensus.cc:2229] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:35.182389 2727 ts_tablet_manager.cc:1905] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.182391 2729 raft_consensus.cc:2258] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.183261 2729 ts_tablet_manager.cc:1905] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.188673 2728 ts_tablet_manager.cc:1918] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:35.188866 2728 log.cc:1196] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/5d861bfac35f49578910db210f17343b
I20260620 06:57:35.189389 2728 ts_tablet_manager.cc:1939] T 5d861bfac35f49578910db210f17343b P 0df52c0020284257bb59594c9e4423b9: Deleting consensus metadata
I20260620 06:57:35.189352 2727 ts_tablet_manager.cc:1918] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:35.189524 2727 log.cc:1196] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/5d861bfac35f49578910db210f17343b
I20260620 06:57:35.189983 2727 ts_tablet_manager.cc:1939] T 5d861bfac35f49578910db210f17343b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting consensus metadata
I20260620 06:57:35.189998 2729 ts_tablet_manager.cc:1918] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:35.190104 2729 log.cc:1196] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/5d861bfac35f49578910db210f17343b
I20260620 06:57:35.190583 2729 ts_tablet_manager.cc:1939] T 5d861bfac35f49578910db210f17343b P fc304b0465f245049586d526b461692c: Deleting consensus metadata
I20260620 06:57:35.190588 2126 catalog_manager.cc:4841] TS 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963): tablet 5d861bfac35f49578910db210f17343b (table test-table [id=02ea60337e334ed0bb43c981dcb45f88]) successfully deleted
I20260620 06:57:35.191203 2128 catalog_manager.cc:4841] TS ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245): tablet 5d861bfac35f49578910db210f17343b (table test-table [id=02ea60337e334ed0bb43c981dcb45f88]) successfully deleted
I20260620 06:57:35.191618 2128 catalog_manager.cc:4841] TS fc304b0465f245049586d526b461692c (127.0.97.3:42449): tablet 5d861bfac35f49578910db210f17343b (table test-table [id=02ea60337e334ed0bb43c981dcb45f88]) successfully deleted
Jun 20 06:57:35 dist-test-slave-mz20 krb5kdc[2096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:57:35 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:35.218968 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.205873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.97.62:39027)
0620 06:57:35.206459 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.206466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.206563 (+ 97us) server_negotiation.cc:407] Connection header received
0620 06:57:35.206753 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.206758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.206831 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.206921 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:35.208166 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.208909 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.210118 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.210482 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.210623 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.214281 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:35.214286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:35.214290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:35.214344 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:35.216832 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:35.217423 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:35.217427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:35.217429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:35.217536 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:35.217792 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:35.217795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:35.217797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:35.218051 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:35.218250 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.218445 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.218676 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":286,"threads_started":1}
I20260620 06:57:35.222034 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:35.222633 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:35.228451 2278 tablet_service.cc:1467] Processing CreateTablet for tablet f13e11ae52b442399f1027a4906c9a4b (DEFAULT_TABLE table=test-table [id=113397ed0d894b4c8ab855cb00698366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.228571 2417 tablet_service.cc:1467] Processing CreateTablet for tablet f13e11ae52b442399f1027a4906c9a4b (DEFAULT_TABLE table=test-table [id=113397ed0d894b4c8ab855cb00698366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.228883 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13e11ae52b442399f1027a4906c9a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.228956 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13e11ae52b442399f1027a4906c9a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.229362 2556 tablet_service.cc:1467] Processing CreateTablet for tablet f13e11ae52b442399f1027a4906c9a4b (DEFAULT_TABLE table=test-table [id=113397ed0d894b4c8ab855cb00698366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.229732 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f13e11ae52b442399f1027a4906c9a4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.234931 2738 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Bootstrap starting.
I20260620 06:57:35.236243 2738 tablet_bootstrap.cc:654] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.236464 2740 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Bootstrap starting.
I20260620 06:57:35.238813 2740 tablet_bootstrap.cc:654] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.240509 2739 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Bootstrap starting.
I20260620 06:57:35.240967 2740 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: No bootstrap required, opened a new log
I20260620 06:57:35.241101 2740 ts_tablet_manager.cc:1397] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:35.241822 2738 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: No bootstrap required, opened a new log
I20260620 06:57:35.241814 2740 raft_consensus.cc:348] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.242040 2738 ts_tablet_manager.cc:1397] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:57:35.242053 2739 tablet_bootstrap.cc:654] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.242151 2740 raft_consensus.cc:374] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.242278 2740 raft_consensus.cc:729] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc304b0465f245049586d526b461692c, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.242583 2740 consensus_queue.cc:260] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.242780 2738 raft_consensus.cc:348] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.243055 2738 raft_consensus.cc:374] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.243182 2738 raft_consensus.cc:729] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.243266 2740 ts_tablet_manager.cc:1428] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:57:35.243618 2738 consensus_queue.cc:260] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.244971 2738 ts_tablet_manager.cc:1428] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:35.245266 2739 tablet_bootstrap.cc:492] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: No bootstrap required, opened a new log
I20260620 06:57:35.245415 2739 ts_tablet_manager.cc:1397] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:57:35.246146 2739 raft_consensus.cc:348] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.246428 2739 raft_consensus.cc:374] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.246585 2739 raft_consensus.cc:729] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.246860 2739 consensus_queue.cc:260] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.247715 2739 ts_tablet_manager.cc:1428] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 06:57:35.249130 2745 raft_consensus.cc:482] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:35.249387 2745 raft_consensus.cc:504] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.251017 2745 leader_election.cc:288] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:35.262880 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.251250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.97.1:52489)
0620 06:57:35.256295 (+ 5045us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:35.256310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:35.256471 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:35.256754 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:35.256759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:35.256780 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:35.257108 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.257119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.258520 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.258523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.260767 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.260782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.262182 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.262186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.262319 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.262343 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:57:35.262410 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:57:35.262517 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4951,"mutex_wait_us":76,"thread_start_us":4805,"threads_started":1}
I20260620 06:57:35.262909 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.251575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52489 (local address 127.0.97.3:42449)
0620 06:57:35.252095 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.252100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.256380 (+ 4280us) server_negotiation.cc:407] Connection header received
0620 06:57:35.256588 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.256593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.256653 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.256731 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:35.257285 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.258398 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.260919 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.262026 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.262165 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.262266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.262450 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.262598 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260620 06:57:35.263460 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13e11ae52b442399f1027a4906c9a4b" candidate_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c" is_pre_election: true
I20260620 06:57:35.263590 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.251257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56717 (local address 127.0.97.2:32963)
0620 06:57:35.256714 (+ 5457us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.256718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.256736 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:57:35.256784 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.256787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.256839 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.256913 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:35.257463 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.258524 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.261352 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.262577 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.262706 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.262821 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.263153 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.263280 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5363,"thread_start_us":5244,"threads_started":1}
I20260620 06:57:35.263598 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.251196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32963 (local address 127.0.97.1:56717)
0620 06:57:35.251678 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:35.251693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:35.251798 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:35.256929 (+ 5131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:35.256934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:35.256958 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:35.257327 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.257337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.258655 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.258659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.261235 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.261244 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.262782 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.262788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.262951 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.262993 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 06:57:35.263114 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 06:57:35.263271 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":9,"thread_start_us":248,"threads_started":1}
I20260620 06:57:35.263675 2575 raft_consensus.cc:2454] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce8abf9305214ecdb778167a1a4cd3ad in term 0.
I20260620 06:57:35.264114 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13e11ae52b442399f1027a4906c9a4b" candidate_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9" is_pre_election: true
I20260620 06:57:35.264168 2231 leader_election.cc:302] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce8abf9305214ecdb778167a1a4cd3ad, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:35.264294 2436 raft_consensus.cc:2454] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce8abf9305214ecdb778167a1a4cd3ad in term 0.
I20260620 06:57:35.264497 2745 raft_consensus.cc:2781] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:35.264604 2745 raft_consensus.cc:482] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:35.264698 2745 raft_consensus.cc:3037] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.266808 2745 raft_consensus.cc:504] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.267316 2745 leader_election.cc:288] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [CANDIDATE]: Term 1 election: Requested vote from peers 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:35.267702 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13e11ae52b442399f1027a4906c9a4b" candidate_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0df52c0020284257bb59594c9e4423b9"
I20260620 06:57:35.267814 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f13e11ae52b442399f1027a4906c9a4b" candidate_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c"
I20260620 06:57:35.267887 2436 raft_consensus.cc:3037] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.268008 2575 raft_consensus.cc:3037] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.270695 2436 raft_consensus.cc:2454] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce8abf9305214ecdb778167a1a4cd3ad in term 1.
I20260620 06:57:35.270696 2575 raft_consensus.cc:2454] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce8abf9305214ecdb778167a1a4cd3ad in term 1.
I20260620 06:57:35.271049 2231 leader_election.cc:302] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, ce8abf9305214ecdb778167a1a4cd3ad, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:35.271328 2745 raft_consensus.cc:2781] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:35.271708 2745 raft_consensus.cc:686] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 LEADER]: Becoming Leader. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Running, Role: LEADER
I20260620 06:57:35.272054 2745 consensus_queue.cc:237] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.275280 2141 catalog_manager.cc:5495] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1). New cstate: current_term: 1 leader_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } health_report { overall_health: UNKNOWN } } }
I20260620 06:57:35.295890 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.290593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.97.1:38245)
0620 06:57:35.291183 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.291187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.291211 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:35.291290 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.291293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.291368 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.291510 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:35.291844 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.292637 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.293836 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.294251 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.294337 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.294810 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.295363 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.295469 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.295574 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.295643 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":339,"threads_started":1}
I20260620 06:57:35.310734 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.305139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58149 (local address 127.0.97.3:42449)
0620 06:57:35.305289 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.305293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.306118 (+ 825us) server_negotiation.cc:407] Connection header received
0620 06:57:35.306279 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.306282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.306349 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.306429 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:35.306915 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.307670 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.309131 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.309537 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.309633 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.310056 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.310154 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.310214 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.310407 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.310477 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:57:35.310784 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.304909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42449 (local address 127.0.97.1:58149)
0620 06:57:35.306029 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:35.306044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:35.306153 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:35.306438 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:35.306442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:35.306454 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:35.306786 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.306795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.307785 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.307789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.309001 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.309014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.309716 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.309721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.309855 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.309912 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.310248 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.310267 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:57:35.310336 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:57:35.310444 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":800,"threads_started":1}
I20260620 06:57:35.318939 2575 raft_consensus.cc:1261] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Refusing update from remote peer ce8abf9305214ecdb778167a1a4cd3ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:35.319002 2436 raft_consensus.cc:1261] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Refusing update from remote peer ce8abf9305214ecdb778167a1a4cd3ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:35.319584 2750 consensus_queue.cc:1035] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:35.319960 2745 consensus_queue.cc:1035] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:35.344276 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.339215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42805 (local address 127.0.97.1:38245)
0620 06:57:35.339437 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.339443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.339706 (+ 263us) server_negotiation.cc:407] Connection header received
0620 06:57:35.339831 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.339834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.339893 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.339966 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:35.340484 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.341331 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.342815 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.343214 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.343305 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.343642 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.343744 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.343826 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.343938 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.343999 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:57:35.344305 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.339077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38245 (local address 127.0.97.3:42805)
0620 06:57:35.339625 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:35.339644 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:35.339744 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:35.339992 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:35.339996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:35.340007 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:57:35.340277 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.340288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.341459 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.341464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.342694 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:35.342706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.343348 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.343352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:35.343463 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.343522 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.343829 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.343838 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:57:35.343913 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:57:35.344042 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":301,"threads_started":1}
I20260620 06:57:35.375468 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:35.375715 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:35.378321 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Sending DeleteTablet for 3 replicas of tablet f13e11ae52b442399f1027a4906c9a4b
I20260620 06:57:35.378995 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet f13e11ae52b442399f1027a4906c9a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53704
I20260620 06:57:35.379138 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet f13e11ae52b442399f1027a4906c9a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52166
I20260620 06:57:35.379283 2727 tablet_replica.cc:330] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: stopping tablet replica
I20260620 06:57:35.379400 2728 tablet_replica.cc:330] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: stopping tablet replica
I20260620 06:57:35.379473 2727 raft_consensus.cc:2229] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:35.379590 2728 raft_consensus.cc:2229] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:35.379777 2728 raft_consensus.cc:2258] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.379963 2727 raft_consensus.cc:2258] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.380139 2141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:57:35.380458 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet f13e11ae52b442399f1027a4906c9a4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58840
W20260620 06:57:35.380702 2141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:35.380734 2729 tablet_replica.cc:330] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: stopping tablet replica
I20260620 06:57:35.380923 2729 raft_consensus.cc:2229] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:35.381085 2729 raft_consensus.cc:2258] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:35.381382 2727 ts_tablet_manager.cc:1905] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.382164 2729 ts_tablet_manager.cc:1905] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.386735 2728 ts_tablet_manager.cc:1905] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:35.388011 2727 ts_tablet_manager.cc:1918] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:35.388150 2727 log.cc:1196] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/f13e11ae52b442399f1027a4906c9a4b
I20260620 06:57:35.388526 2727 ts_tablet_manager.cc:1939] T f13e11ae52b442399f1027a4906c9a4b P ce8abf9305214ecdb778167a1a4cd3ad: Deleting consensus metadata
I20260620 06:57:35.389596 2128 catalog_manager.cc:4841] TS ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245): tablet f13e11ae52b442399f1027a4906c9a4b (table test-table [id=113397ed0d894b4c8ab855cb00698366]) successfully deleted
I20260620 06:57:35.391193 2729 ts_tablet_manager.cc:1918] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:35.391434 2729 log.cc:1196] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/f13e11ae52b442399f1027a4906c9a4b
I20260620 06:57:35.391899 2729 ts_tablet_manager.cc:1939] T f13e11ae52b442399f1027a4906c9a4b P fc304b0465f245049586d526b461692c: Deleting consensus metadata
I20260620 06:57:35.393042 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 (DEFAULT_TABLE table=test-table [id=da3ea7e45622478493c9a2d2c06ba065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.393465 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759a886bfa1f4b82b21980a1c58e6a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.393437 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 (DEFAULT_TABLE table=test-table [id=da3ea7e45622478493c9a2d2c06ba065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.393447 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 (DEFAULT_TABLE table=test-table [id=da3ea7e45622478493c9a2d2c06ba065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:35.393821 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759a886bfa1f4b82b21980a1c58e6a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.393841 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 759a886bfa1f4b82b21980a1c58e6a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:35.398727 2128 catalog_manager.cc:4841] TS fc304b0465f245049586d526b461692c (127.0.97.3:42449): tablet f13e11ae52b442399f1027a4906c9a4b (table test-table [id=113397ed0d894b4c8ab855cb00698366]) successfully deleted
I20260620 06:57:35.399937 2740 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Bootstrap starting.
I20260620 06:57:35.401389 2739 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Bootstrap starting.
I20260620 06:57:35.401507 2740 tablet_bootstrap.cc:654] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.401832 2728 ts_tablet_manager.cc:1918] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:57:35.401975 2728 log.cc:1196] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/f13e11ae52b442399f1027a4906c9a4b
I20260620 06:57:35.402369 2728 ts_tablet_manager.cc:1939] T f13e11ae52b442399f1027a4906c9a4b P 0df52c0020284257bb59594c9e4423b9: Deleting consensus metadata
I20260620 06:57:35.403131 2739 tablet_bootstrap.cc:654] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.403719 2126 catalog_manager.cc:4841] TS 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963): tablet f13e11ae52b442399f1027a4906c9a4b (table test-table [id=113397ed0d894b4c8ab855cb00698366]) successfully deleted
I20260620 06:57:35.404646 2738 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Bootstrap starting.
I20260620 06:57:35.406211 2738 tablet_bootstrap.cc:654] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:35.411226 2738 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: No bootstrap required, opened a new log
I20260620 06:57:35.411410 2738 ts_tablet_manager.cc:1397] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 06:57:35.411582 2740 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: No bootstrap required, opened a new log
I20260620 06:57:35.411741 2740 ts_tablet_manager.cc:1397] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260620 06:57:35.411998 2739 tablet_bootstrap.cc:492] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: No bootstrap required, opened a new log
I20260620 06:57:35.412150 2739 ts_tablet_manager.cc:1397] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 06:57:35.412192 2738 raft_consensus.cc:348] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.412441 2738 raft_consensus.cc:374] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.412513 2738 raft_consensus.cc:729] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.412541 2740 raft_consensus.cc:348] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.412767 2740 raft_consensus.cc:374] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.412834 2740 raft_consensus.cc:729] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc304b0465f245049586d526b461692c, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.412739 2738 consensus_queue.cc:260] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.412879 2739 raft_consensus.cc:348] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.413101 2739 raft_consensus.cc:374] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:35.413170 2739 raft_consensus.cc:729] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce8abf9305214ecdb778167a1a4cd3ad, State: Initialized, Role: FOLLOWER
I20260620 06:57:35.413075 2740 consensus_queue.cc:260] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.413269 2738 ts_tablet_manager.cc:1428] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:57:35.413574 2740 ts_tablet_manager.cc:1428] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 06:57:35.413921 2739 consensus_queue.cc:260] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.414688 2739 ts_tablet_manager.cc:1428] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 06:57:35.434944 2744 raft_consensus.cc:482] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:35.435169 2744 raft_consensus.cc:504] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.435901 2744 leader_election.cc:288] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:35.436367 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a886bfa1f4b82b21980a1c58e6a37" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" is_pre_election: true
I20260620 06:57:35.436398 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a886bfa1f4b82b21980a1c58e6a37" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c" is_pre_election: true
I20260620 06:57:35.436611 2576 raft_consensus.cc:2454] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 0.
I20260620 06:57:35.436616 2298 raft_consensus.cc:2454] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 0.
I20260620 06:57:35.436993 2371 leader_election.cc:302] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:35.437259 2744 raft_consensus.cc:2781] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:35.437372 2744 raft_consensus.cc:482] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:35.437458 2744 raft_consensus.cc:3037] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.439546 2744 raft_consensus.cc:504] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.440021 2744 leader_election.cc:288] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 election: Requested vote from peers ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245), fc304b0465f245049586d526b461692c (127.0.97.3:42449)
I20260620 06:57:35.440537 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a886bfa1f4b82b21980a1c58e6a37" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc304b0465f245049586d526b461692c"
I20260620 06:57:35.440585 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a886bfa1f4b82b21980a1c58e6a37" candidate_uuid: "0df52c0020284257bb59594c9e4423b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce8abf9305214ecdb778167a1a4cd3ad"
I20260620 06:57:35.440738 2576 raft_consensus.cc:3037] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.440766 2298 raft_consensus.cc:3037] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:35.443524 2576 raft_consensus.cc:2454] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 1.
I20260620 06:57:35.443655 2298 raft_consensus.cc:2454] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0df52c0020284257bb59594c9e4423b9 in term 1.
I20260620 06:57:35.444026 2371 leader_election.cc:302] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0df52c0020284257bb59594c9e4423b9, fc304b0465f245049586d526b461692c; no voters:
I20260620 06:57:35.444303 2744 raft_consensus.cc:2781] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:35.444725 2744 raft_consensus.cc:686] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 1 LEADER]: Becoming Leader. State: Replica: 0df52c0020284257bb59594c9e4423b9, State: Running, Role: LEADER
I20260620 06:57:35.445014 2744 consensus_queue.cc:237] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } }
I20260620 06:57:35.447511 2138 catalog_manager.cc:5495] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0df52c0020284257bb59594c9e4423b9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0df52c0020284257bb59594c9e4423b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0df52c0020284257bb59594c9e4423b9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 } health_report { overall_health: UNKNOWN } } }
I20260620 06:57:35.471417 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:35.464933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.0.97.2:32963)
0620 06:57:35.465140 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:35.465144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:35.465188 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:57:35.465405 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:35.465408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:35.465465 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:35.465540 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:35.465913 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.466661 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.468540 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:35.469171 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:35.469284 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:35.470375 (+ 1091us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:35.470524 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:35.470623 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:35.471067 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 06:57:35.471144 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 06:57:35.480513 2576 raft_consensus.cc:1261] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Refusing update from remote peer 0df52c0020284257bb59594c9e4423b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:35.480886 2298 raft_consensus.cc:1261] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Refusing update from remote peer 0df52c0020284257bb59594c9e4423b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:35.480985 2767 consensus_queue.cc:1035] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc304b0465f245049586d526b461692c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:35.481524 2767 consensus_queue.cc:1035] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce8abf9305214ecdb778167a1a4cd3ad" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:57:35.502288 2710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:57:37.083669 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:37.083925 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:37.088313 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64e1c36822a64b0b8cafe9ed0d21cd03: Sending DeleteTablet for 3 replicas of tablet 759a886bfa1f4b82b21980a1c58e6a37
I20260620 06:57:37.089048 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52166
I20260620 06:57:37.089253 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53704
I20260620 06:57:37.089284 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a886bfa1f4b82b21980a1c58e6a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58840
I20260620 06:57:37.090166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2212
I20260620 06:57:37.100486 2800 tablet_replica.cc:330] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: stopping tablet replica
I20260620 06:57:37.100677 2800 raft_consensus.cc:2229] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:37.100870 2800 raft_consensus.cc:2258] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:37.101660 2800 ts_tablet_manager.cc:1905] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:37.101691 2799 tablet_replica.cc:330] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: stopping tablet replica
I20260620 06:57:37.101882 2799 raft_consensus.cc:2229] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:37.102252 2799 raft_consensus.cc:2258] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:37.102818 2801 tablet_replica.cc:330] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: stopping tablet replica
I20260620 06:57:37.103127 2801 raft_consensus.cc:2229] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:37.103420 2801 raft_consensus.cc:2258] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:37.103569 2799 ts_tablet_manager.cc:1905] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:37.104472 2801 ts_tablet_manager.cc:1905] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:37.108361 2800 ts_tablet_manager.cc:1918] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:37.108592 2800 log.cc:1196] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/759a886bfa1f4b82b21980a1c58e6a37
I20260620 06:57:37.109103 2800 ts_tablet_manager.cc:1939] T 759a886bfa1f4b82b21980a1c58e6a37 P ce8abf9305214ecdb778167a1a4cd3ad: Deleting consensus metadata
I20260620 06:57:37.110374 2128 catalog_manager.cc:4841] TS ce8abf9305214ecdb778167a1a4cd3ad (127.0.97.1:38245): tablet 759a886bfa1f4b82b21980a1c58e6a37 (table test-table [id=da3ea7e45622478493c9a2d2c06ba065]) successfully deleted
I20260620 06:57:37.110592 2799 ts_tablet_manager.cc:1918] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:37.110760 2799 log.cc:1196] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/759a886bfa1f4b82b21980a1c58e6a37
I20260620 06:57:37.111101 2799 ts_tablet_manager.cc:1939] T 759a886bfa1f4b82b21980a1c58e6a37 P 0df52c0020284257bb59594c9e4423b9: Deleting consensus metadata
I20260620 06:57:37.111421 2801 ts_tablet_manager.cc:1918] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:57:37.111555 2801 log.cc:1196] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/759a886bfa1f4b82b21980a1c58e6a37
I20260620 06:57:37.111961 2801 ts_tablet_manager.cc:1939] T 759a886bfa1f4b82b21980a1c58e6a37 P fc304b0465f245049586d526b461692c: Deleting consensus metadata
I20260620 06:57:37.112018 2126 catalog_manager.cc:4841] TS 0df52c0020284257bb59594c9e4423b9 (127.0.97.2:32963): tablet 759a886bfa1f4b82b21980a1c58e6a37 (table test-table [id=da3ea7e45622478493c9a2d2c06ba065]) successfully deleted
I20260620 06:57:37.112895 2128 catalog_manager.cc:4841] TS fc304b0465f245049586d526b461692c (127.0.97.3:42449): tablet 759a886bfa1f4b82b21980a1c58e6a37 (table test-table [id=da3ea7e45622478493c9a2d2c06ba065]) successfully deleted
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:37.126210 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:37.091358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.0.97.1:38245)
0620 06:57:37.097114 (+ 5756us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:37.097121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:37.107010 (+ 9889us) server_negotiation.cc:407] Connection header received
0620 06:57:37.107518 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:37.107524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:37.107630 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:37.107718 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:37.110272 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.111375 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.113057 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.113453 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.113549 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:37.120834 (+ 7285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:37.120850 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:37.120866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:37.120901 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:37.123653 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.124276 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.124284 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.124286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.124416 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.124768 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.124771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.124773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.125113 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:37.125388 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:37.125653 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 06:57:37.125866 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5597,"thread_start_us":5481,"threads_started":1}
W20260620 06:57:37.127230 2343 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53710
W20260620 06:57:37.127839 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:37.128001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2212
I20260620 06:57:37.145360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2352
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:37.158749 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:37.146054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52198 (local address 127.0.97.2:32963)
0620 06:57:37.146492 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:37.146496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:37.146512 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:57:37.146594 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:37.146600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:37.146657 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:37.146747 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:37.147921 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.148633 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.149748 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.150138 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.150233 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:37.153885 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:37.153899 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:37.153914 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:37.153945 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:37.156587 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.157085 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.157091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.157094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.157199 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.157479 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.157482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.157484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.157793 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:37.158003 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:37.158245 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 06:57:37.158406 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
W20260620 06:57:37.159512 2482 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52198
W20260620 06:57:37.160068 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2352: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:37.160216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2352
I20260620 06:57:37.177613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2491
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:37.191771 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:37.178328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.97.3:42449)
0620 06:57:37.178875 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:37.178879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:37.178895 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:57:37.178948 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:37.178952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:37.179009 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:37.179096 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:37.180204 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.180915 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.182051 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:37.182416 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:37.182501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:37.186443 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:37.186458 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:37.186474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:37.186510 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:37.189201 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.189873 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.189880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.189882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.190027 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:37.190431 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:37.190434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:37.190436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:37.190804 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:37.191042 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:37.191282 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:57:37.191436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":292,"threads_started":1}
W20260620 06:57:37.192395 2621 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58874
W20260620 06:57:37.192983 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2491: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:37.193114 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2491
I20260620 06:57:37.213440 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2109
I20260620 06:57:37.214526 2181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:37.325881 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2109
2026-06-20T06:57:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7907 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2809](info): set up 2 sockets
Jun 20 06:57:37 dist-test-slave-mz20 krb5kdc[2809](info): commencing operation
krb5kdc: starting...
W20260620 06:57:38.322531 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:38 dist-test-slave-mz20 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:38.492313 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:38.798997 2822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:38.799307 2822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:38.799436 2822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:38.806771 2822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:38.806910 2822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:38.806991 2822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:38.807063 2822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:38.807126 2822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:38.817087 2822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:38.818468 2822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:38.819778 2822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:38.830894 2827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:38.831002 2828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:38.830925 2830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:38.831609 2822 server_base.cc:1034] running on GCE node
I20260620 06:57:38.832333 2822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:38.833653 2822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:38.834839 2822 hybrid_clock.cc:648] HybridClock initialized: now 1781938658834797 us; error 32 us; skew 500 ppm
I20260620 06:57:38.835183 2822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:38 dist-test-slave-mz20 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:38.838171 2822 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:38.839500 2822 webserver.cc:460] Webserver started at http://127.0.97.62:39927/ using document root <none> and password file <none>
I20260620 06:57:38.840147 2822 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:38.840265 2822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:38.840550 2822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:38.843577 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d"
format_stamp: "Formatted at 2026-06-20 06:57:38 on dist-test-slave-mz20"
server_key: "ffd8312e8d2bd187d73b2f1f97f451a8"
server_key_iv: "beeaf19ab3c51392141db296d19540e7"
server_key_version: "encryptionkey@0"
I20260620 06:57:38.844295 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d"
format_stamp: "Formatted at 2026-06-20 06:57:38 on dist-test-slave-mz20"
server_key: "ffd8312e8d2bd187d73b2f1f97f451a8"
server_key_iv: "beeaf19ab3c51392141db296d19540e7"
server_key_version: "encryptionkey@0"
I20260620 06:57:38.849095 2822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:57:38.852267 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:38.853493 2822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 06:57:38.853634 2822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d"
format_stamp: "Formatted at 2026-06-20 06:57:38 on dist-test-slave-mz20"
server_key: "ffd8312e8d2bd187d73b2f1f97f451a8"
server_key_iv: "beeaf19ab3c51392141db296d19540e7"
server_key_version: "encryptionkey@0"
I20260620 06:57:38.853798 2822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:38.882931 2822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:38.886608 2822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:38.886855 2822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:38.903204 2822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46209
I20260620 06:57:38.903234 2888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46209 every 8 connection(s)
I20260620 06:57:38.904513 2822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:38.908116 2889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:38.908679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2822
I20260620 06:57:38.908846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:38.909435 388 external_mini_cluster.cc:1428] Setting key d5f21b04a701fbadfd110535bdde7b82
I20260620 06:57:38.921767 2889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d: Bootstrap starting.
Jun 20 06:57:38 dist-test-slave-mz20 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:38.924767 2889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:38.926106 2889 log.cc:824] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:38.929420 2889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d: No bootstrap required, opened a new log
I20260620 06:57:38.932859 2892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:38.911391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.0.97.62:46209)
0620 06:57:38.912139 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:38.912154 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:38.912204 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:57:38.914208 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:38.914237 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:38.914803 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:38.915261 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:38.916482 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:38.918379 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:38.919665 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:38.920721 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:38.920896 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:38.924815 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:38.924826 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:38.924846 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:38.924893 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:38.928393 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:38.929100 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:38.929107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:38.929115 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:38.929314 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:38.929683 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:38.929688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:38.929691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:38.930398 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:38.930724 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:38.931051 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 06:57:38.931501 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":289,"threads_started":1}
I20260620 06:57:38.937867 2889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } }
I20260620 06:57:38.938223 2889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:38.938301 2889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6da0c1318e4d65ad8f71b337ed1b4d, State: Initialized, Role: FOLLOWER
I20260620 06:57:38.938840 2889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } }
I20260620 06:57:38.939055 2889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:38.939173 2889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:38.939328 2889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:38.941502 2889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } }
I20260620 06:57:38.941920 2889 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6da0c1318e4d65ad8f71b337ed1b4d; no voters:
I20260620 06:57:38.942471 2889 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:38.942624 2894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:38.943238 2894 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [term 1 LEADER]: Becoming Leader. State: Replica: 8f6da0c1318e4d65ad8f71b337ed1b4d, State: Running, Role: LEADER
I20260620 06:57:38.943748 2894 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } }
I20260620 06:57:38.944278 2889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:38.946625 2895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } } }
I20260620 06:57:38.947316 2895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:38.947052 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6da0c1318e4d65ad8f71b337ed1b4d. Latest consensus state: current_term: 1 leader_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6da0c1318e4d65ad8f71b337ed1b4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46209 } } }
I20260620 06:57:38.947758 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:38.948786 2903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:38.953042 2903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:38.961632 2903 catalog_manager.cc:1347] Generated new cluster ID: 63901bd1cd0b4429b6e1b6cb82b485ae
I20260620 06:57:38.961784 2903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:38.974660 2903 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:38.976500 2903 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:38.990036 2903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6da0c1318e4d65ad8f71b337ed1b4d: Generated new TSK 0
I20260620 06:57:38.991178 2903 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:39.033968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:39.344748 2915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:39.345047 2915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:39.345177 2915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:39.352458 2915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:39.352569 2915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:39.352715 2915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:39.362246 2915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:39.363610 2915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:39.364849 2915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:39.376884 2920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:39.376914 2923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:39.376937 2921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:39.377655 2915 server_base.cc:1034] running on GCE node
I20260620 06:57:39.378207 2915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:39.379117 2915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:39.380311 2915 hybrid_clock.cc:648] HybridClock initialized: now 1781938659380258 us; error 48 us; skew 500 ppm
I20260620 06:57:39.380596 2915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:39 dist-test-slave-mz20 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:39.383477 2915 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:39.384727 2915 webserver.cc:460] Webserver started at http://127.0.97.1:41651/ using document root <none> and password file <none>
I20260620 06:57:39.385366 2915 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:39.385476 2915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:39.385751 2915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:39.388491 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "d081683804694ef3bee46f66966c22d3"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "1db14e33884eb17e5dc4cf4ed6b70811"
server_key_iv: "362f9f57251f76aedfb0b9d41d67074f"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.389221 2915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d081683804694ef3bee46f66966c22d3"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "1db14e33884eb17e5dc4cf4ed6b70811"
server_key_iv: "362f9f57251f76aedfb0b9d41d67074f"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.394305 2915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 06:57:39.397650 2930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.399029 2915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:39.399175 2915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d081683804694ef3bee46f66966c22d3"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "1db14e33884eb17e5dc4cf4ed6b70811"
server_key_iv: "362f9f57251f76aedfb0b9d41d67074f"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.399354 2915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:39.412259 2915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:39.416601 2915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:39.416859 2915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:39.417802 2915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:39.419209 2915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:39.419286 2915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.419404 2915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:39.419459 2915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.448200 2915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39485
I20260620 06:57:39.448282 3042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39485 every 8 connection(s)
I20260620 06:57:39.449455 2915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:39.451299 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 2915
I20260620 06:57:39.451490 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:39.452265 388 external_mini_cluster.cc:1428] Setting key 379b6419a2649b5477eee564fc9d223b
Jun 20 06:57:39 dist-test-slave-mz20 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:39.470721 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:39.451886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52503 (local address 127.0.97.62:46209)
0620 06:57:39.452322 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:39.452328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:39.453508 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 06:57:39.455517 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:39.455523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:39.455600 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:39.455709 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:39.457106 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.458082 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.460228 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.460649 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.460738 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:39.464547 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:39.464553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:39.464559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:39.464599 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:39.467039 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:39.467950 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:39.467956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:39.467960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:39.468106 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:39.468686 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:39.468692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:39.468696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:39.469037 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:39.469182 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:39.470074 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 06:57:39.470324 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":242,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 06:57:39.471938 3046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:39.452549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46209 (local address 127.0.97.1:52503)
0620 06:57:39.453334 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:39.453391 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:39.455244 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:39.455885 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:39.455893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:39.456379 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:39.456951 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:39.456968 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.458232 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.458236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:39.460082 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:39.460096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.460849 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.460855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:39.461124 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:39.461775 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:39.461801 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:39.464312 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:39.467194 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:39.467203 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:39.467221 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:39.467801 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:39.468240 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:39.468245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:39.468248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:39.468521 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:39.469187 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:39.469196 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:39.469650 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 06:57:39.469964 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 06:57:39.470498 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
I20260620 06:57:39.474375 3043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46209
I20260620 06:57:39.474740 3043 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:39.475595 3043 heartbeater.cc:511] Master 127.0.97.62:46209 requested a full tablet report, sending...
I20260620 06:57:39.477774 2854 ts_manager.cc:194] Registered new tserver with Master: d081683804694ef3bee46f66966c22d3 (127.0.97.1:39485)
I20260620 06:57:39.479269 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:39.493407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:39.800226 3049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:39.800485 3049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:39.800593 3049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:39.807585 3049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:39.807696 3049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:39.807857 3049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:39.817453 3049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:39.818738 3049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:39.820048 3049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:39.832159 3055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:39.832188 3054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:39.832132 3057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:39.832887 3049 server_base.cc:1034] running on GCE node
I20260620 06:57:39.833422 3049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:39.834241 3049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:39.835431 3049 hybrid_clock.cc:648] HybridClock initialized: now 1781938659835379 us; error 47 us; skew 500 ppm
I20260620 06:57:39.835726 3049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:39 dist-test-slave-mz20 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:39.838497 3049 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:39.839771 3049 webserver.cc:460] Webserver started at http://127.0.97.2:34581/ using document root <none> and password file <none>
I20260620 06:57:39.840379 3049 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:39.840479 3049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:39.840719 3049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:39.843915 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "87b9d3ef31ba42b7a57f9e6130c06409"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "3c0c9bbfd2130d10e55670985bc87b75"
server_key_iv: "b9eb76065d5eb11589a65a1225e65536"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.844885 3049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "87b9d3ef31ba42b7a57f9e6130c06409"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "3c0c9bbfd2130d10e55670985bc87b75"
server_key_iv: "b9eb76065d5eb11589a65a1225e65536"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.851092 3049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 06:57:39.854282 3064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.855544 3049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:39.855703 3049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "87b9d3ef31ba42b7a57f9e6130c06409"
format_stamp: "Formatted at 2026-06-20 06:57:39 on dist-test-slave-mz20"
server_key: "3c0c9bbfd2130d10e55670985bc87b75"
server_key_iv: "b9eb76065d5eb11589a65a1225e65536"
server_key_version: "encryptionkey@0"
I20260620 06:57:39.855888 3049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:39.870100 3049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:39.873610 3049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:39.873842 3049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:39.874702 3049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:39.875873 3049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:39.875944 3049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.876053 3049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:39.876106 3049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:39.904524 3049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45341
I20260620 06:57:39.904546 3176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45341 every 8 connection(s)
I20260620 06:57:39.905783 3049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:39.910130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3049
I20260620 06:57:39.910290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:39.910753 388 external_mini_cluster.cc:1428] Setting key 1626b195f839273acf7c5ab271e2515f
Jun 20 06:57:39 dist-test-slave-mz20 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:39.928872 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:39.908397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35285 (local address 127.0.97.62:46209)
0620 06:57:39.908595 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:39.908601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:39.909910 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 06:57:39.912510 (+ 2600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:39.912515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:39.912626 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:39.912727 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:39.914653 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.915641 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.917962 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.918644 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.918759 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:39.922564 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:39.922571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:39.922576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:39.922624 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:39.925675 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:39.926414 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:39.926420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:39.926423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:39.926635 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:39.927028 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:39.927032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:39.927036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:39.927374 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:39.927486 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:39.928229 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 06:57:39.928467 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 06:57:39.929975 3179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:39.908887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46209 (local address 127.0.97.2:35285)
0620 06:57:39.909716 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:39.909779 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:39.912194 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:39.912982 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:39.912994 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:39.913727 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:39.914453 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:39.914475 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.915839 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.915845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:39.917825 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:39.917839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:39.918850 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:39.918857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:39.919137 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:39.919878 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:39.919907 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:39.922302 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:39.925838 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:39.925846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:39.925860 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:39.926295 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:39.926738 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:39.926741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:39.926744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:39.926926 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:39.927495 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:39.927500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:39.927846 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 06:57:39.928127 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 06:57:39.928683 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":286,"threads_started":1}
I20260620 06:57:39.931504 3177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46209
I20260620 06:57:39.931851 3177 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:39.932716 3177 heartbeater.cc:511] Master 127.0.97.62:46209 requested a full tablet report, sending...
I20260620 06:57:39.934260 2854 ts_manager.cc:194] Registered new tserver with Master: 87b9d3ef31ba42b7a57f9e6130c06409 (127.0.97.2:45341)
I20260620 06:57:39.935086 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:39.951736 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:40.274492 3182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:40.274832 3182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:40.274927 3182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:40.282099 3182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:40.282228 3182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:40.282341 3182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:40.292054 3182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:40.293254 3182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:40.294471 3182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:40.307334 3187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:40.307360 3188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:40.307597 3190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:40.308223 3182 server_base.cc:1034] running on GCE node
I20260620 06:57:40.308859 3182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:40.309827 3182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:40.311026 3182 hybrid_clock.cc:648] HybridClock initialized: now 1781938660310969 us; error 42 us; skew 500 ppm
I20260620 06:57:40.311342 3182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:40 dist-test-slave-mz20 krb5kdc[2809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:40.314694 3182 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:40.316226 3182 webserver.cc:460] Webserver started at http://127.0.97.3:42459/ using document root <none> and password file <none>
I20260620 06:57:40.316962 3182 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:40.317116 3182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:40.317410 3182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:40.320359 3182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "37d8fdbc18ac4b178d328d56420c20d7"
format_stamp: "Formatted at 2026-06-20 06:57:40 on dist-test-slave-mz20"
server_key: "387b17e7859c8cc98ee2f434c9a4c163"
server_key_iv: "566d5155b058e59c6c150e04db329db4"
server_key_version: "encryptionkey@0"
I20260620 06:57:40.321122 3182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "37d8fdbc18ac4b178d328d56420c20d7"
format_stamp: "Formatted at 2026-06-20 06:57:40 on dist-test-slave-mz20"
server_key: "387b17e7859c8cc98ee2f434c9a4c163"
server_key_iv: "566d5155b058e59c6c150e04db329db4"
server_key_version: "encryptionkey@0"
I20260620 06:57:40.326012 3182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:40.329407 3197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:40.330906 3182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:40.331060 3182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "37d8fdbc18ac4b178d328d56420c20d7"
format_stamp: "Formatted at 2026-06-20 06:57:40 on dist-test-slave-mz20"
server_key: "387b17e7859c8cc98ee2f434c9a4c163"
server_key_iv: "566d5155b058e59c6c150e04db329db4"
server_key_version: "encryptionkey@0"
I20260620 06:57:40.331254 3182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:40.345345 3182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:40.349298 3182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:40.349540 3182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:40.350450 3182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:40.351758 3182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:40.351827 3182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:40.351930 3182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:40.351981 3182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:40.379801 3182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35495
I20260620 06:57:40.379843 3309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35495 every 8 connection(s)
I20260620 06:57:40.380999 3182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:40.387982 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3182
I20260620 06:57:40.388154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:40.388682 388 external_mini_cluster.cc:1428] Setting key 12513dcdafb6a6e3a4c8de1ee38eeb49
Jun 20 06:57:40 dist-test-slave-mz20 krb5kdc[2809](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:40.400027 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:40.383381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41525 (local address 127.0.97.62:46209)
0620 06:57:40.383557 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:40.383562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:40.384671 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 06:57:40.386376 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:40.386381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:40.386446 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:40.386570 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:40.388055 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:40.389110 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:40.391120 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:40.391715 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:40.391813 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:40.394984 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:40.394989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:40.394992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:40.395024 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:40.397375 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:40.398022 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:40.398025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:40.398028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:40.398145 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:40.398580 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:40.398584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:40.398588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:40.398880 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:40.398989 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:40.399609 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 06:57:40.399780 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:57:40.400975 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:40.383835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46209 (local address 127.0.97.3:41525)
0620 06:57:40.384514 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:40.384555 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:40.386145 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:40.386812 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:40.386820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:40.387300 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:40.387886 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:40.387903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:40.389269 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:40.389274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:40.390988 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:40.391002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:40.391900 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:40.391906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:40.392102 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:40.392673 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:40.392695 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:40.394786 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:40.397507 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:40.397514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:40.397526 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:40.397921 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:40.398237 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:40.398241 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:40.398243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:40.398463 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:40.398977 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:40.398982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:40.399306 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 06:57:40.399521 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 06:57:40.399902 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260620 06:57:40.402375 3310 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46209
I20260620 06:57:40.402810 3310 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:40.403610 3310 heartbeater.cc:511] Master 127.0.97.62:46209 requested a full tablet report, sending...
I20260620 06:57:40.404997 2854 ts_manager.cc:194] Registered new tserver with Master: 37d8fdbc18ac4b178d328d56420c20d7 (127.0.97.3:35495)
I20260620 06:57:40.405742 2854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41525
I20260620 06:57:40.416180 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:57:40.443248 3045 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:57:40.440955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.0.97.62:46209)
0620 06:57:40.441115 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:40.441120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:40.441536 (+ 416us) server_negotiation.cc:407] Connection header received
0620 06:57:40.441695 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:40.441699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:40.441762 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:40.441853 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:40.443000 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33700: BlockingRecv error: recv got EOF from 127.0.0.1:33700 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:57:40.446192 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2915
W20260620 06:57:40.448830 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2915: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39485: server requires authentication, but client does not have Kerberos credentials available
I20260620 06:57:40.449020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2915
I20260620 06:57:40.461962 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3049
W20260620 06:57:40.464417 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3049: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45341: server requires authentication, but client does not have Kerberos credentials available
I20260620 06:57:40.464525 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3049
I20260620 06:57:40.476218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3182
W20260620 06:57:40.478713 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3182: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:35495: server requires authentication, but client does not have Kerberos credentials available
I20260620 06:57:40.478852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3182
I20260620 06:57:40.490448 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 2822
I20260620 06:57:40.491621 2884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:40.592857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 2822
2026-06-20T06:57:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3264 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:40 dist-test-slave-mz20 krb5kdc[3327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:40 dist-test-slave-mz20 krb5kdc[3327](info): set up 2 sockets
Jun 20 06:57:40 dist-test-slave-mz20 krb5kdc[3327](info): commencing operation
krb5kdc: starting...
W20260620 06:57:41.582593 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:41 dist-test-slave-mz20 krb5kdc[3327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:41.752688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42539
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:42.059918 3340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:42.060194 3340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:42.060288 3340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:42.067332 3340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:42.067432 3340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:42.067476 3340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:42.067531 3340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:42.067576 3340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:42.077018 3340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:42.078171 3340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:42.079468 3340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:42.090713 3345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:42.090780 3346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:42.090809 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:42.091926 3340 server_base.cc:1034] running on GCE node
I20260620 06:57:42.092703 3340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:42.094115 3340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:42.095301 3340 hybrid_clock.cc:648] HybridClock initialized: now 1781938662095252 us; error 48 us; skew 500 ppm
I20260620 06:57:42.095635 3340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:42 dist-test-slave-mz20 krb5kdc[3327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:42.098474 3340 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:42.099846 3340 webserver.cc:460] Webserver started at http://127.0.97.62:36153/ using document root <none> and password file <none>
I20260620 06:57:42.100490 3340 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:42.100601 3340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:42.100865 3340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:42.103710 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "cfd6cbdc6bee4ae38915721e9f53620d"
format_stamp: "Formatted at 2026-06-20 06:57:42 on dist-test-slave-mz20"
server_key: "23dbbd953283c25181a19f683270fe3d"
server_key_iv: "fee43231399de8e4ac2b09ac779f43c8"
server_key_version: "encryptionkey@0"
I20260620 06:57:42.104444 3340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "cfd6cbdc6bee4ae38915721e9f53620d"
format_stamp: "Formatted at 2026-06-20 06:57:42 on dist-test-slave-mz20"
server_key: "23dbbd953283c25181a19f683270fe3d"
server_key_iv: "fee43231399de8e4ac2b09ac779f43c8"
server_key_version: "encryptionkey@0"
I20260620 06:57:42.109522 3340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 06:57:42.113085 3355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:42.114488 3340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:42.114670 3340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "cfd6cbdc6bee4ae38915721e9f53620d"
format_stamp: "Formatted at 2026-06-20 06:57:42 on dist-test-slave-mz20"
server_key: "23dbbd953283c25181a19f683270fe3d"
server_key_iv: "fee43231399de8e4ac2b09ac779f43c8"
server_key_version: "encryptionkey@0"
I20260620 06:57:42.114842 3340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:42.149147 3340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:42.153264 3340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:42.153532 3340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:42.170410 3340 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42539
I20260620 06:57:42.170595 3406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42539 every 8 connection(s)
I20260620 06:57:42.171721 3340 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:42.175297 3407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:42.185503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3340
I20260620 06:57:42.185732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:42.186467 388 external_mini_cluster.cc:1428] Setting key 09f197bf18a9e87bab8bb542185ad417
I20260620 06:57:42.188068 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d: Bootstrap starting.
I20260620 06:57:42.191597 3407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:42.192660 3407 log.cc:824] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:42.195802 3407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d: No bootstrap required, opened a new log
Jun 20 06:57:42 dist-test-slave-mz20 krb5kdc[3327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:42.204532 3407 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 06:57:42.204864 3407 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:42.204963 3407 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd6cbdc6bee4ae38915721e9f53620d, State: Initialized, Role: FOLLOWER
I20260620 06:57:42.205638 3407 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 06:57:42.205912 3407 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:42.206081 3407 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:42.206269 3407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:42.209488 3407 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 06:57:42.209981 3407 leader_election.cc:302] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd6cbdc6bee4ae38915721e9f53620d; no voters:
I20260620 06:57:42.210695 3407 leader_election.cc:288] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:42.210800 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:42.212858 3407 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:42.215396 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:42.189022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.97.62:42539)
0620 06:57:42.189903 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:42.189920 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:42.189974 (+ 54us) server_negotiation.cc:407] Connection header received
0620 06:57:42.192517 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:42.192547 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:42.193060 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:42.193545 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:42.195371 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:42.197358 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:42.199232 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:42.200320 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:42.200504 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:42.205304 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:42.205314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:42.205333 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:42.205381 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:42.209359 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:42.210178 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:42.210186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:42.210209 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:42.210396 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:42.210944 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:42.210949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:42.210951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:42.211704 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:42.211981 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:42.213289 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0620 06:57:42.213726 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":362,"threads_started":1}
I20260620 06:57:42.217886 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [term 1 LEADER]: Becoming Leader. State: Replica: cfd6cbdc6bee4ae38915721e9f53620d, State: Running, Role: LEADER
I20260620 06:57:42.218506 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } }
I20260620 06:57:42.220494 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:57:42.225657 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd6cbdc6bee4ae38915721e9f53620d. Latest consensus state: current_term: 1 leader_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } } }
I20260620 06:57:42.225683 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd6cbdc6bee4ae38915721e9f53620d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42539 } } }
I20260620 06:57:42.226064 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:42.226064 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:42.226966 3424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:42.232995 3424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:42.255911 3424 catalog_manager.cc:1347] Generated new cluster ID: eca5d503bfe3432c815e366d963a4a58
I20260620 06:57:42.256026 3424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:42.272068 3410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:42.265001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.97.62:42539)
0620 06:57:42.265218 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:42.265225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:42.265788 (+ 563us) server_negotiation.cc:407] Connection header received
0620 06:57:42.265962 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:42.265966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:42.266032 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:42.266115 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:42.267111 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:42.267845 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:42.269109 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:42.269535 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:42.269654 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:42.270678 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:42.270683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:42.270686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:42.271305 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:42.271365 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:42.271395 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:42.271489 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:42.271629 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 06:57:42.271728 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:57:42.277846 3424 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:42.279706 3424 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:42.291360 3424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd6cbdc6bee4ae38915721e9f53620d: Generated new TSK 0
I20260620 06:57:42.292217 3424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:57:42.301312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3340
I20260620 06:57:42.302336 3402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:42.391532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3340
2026-06-20T06:57:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1794 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:42 dist-test-slave-mz20 krb5kdc[3441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:42 dist-test-slave-mz20 krb5kdc[3441](info): set up 2 sockets
Jun 20 06:57:42 dist-test-slave-mz20 krb5kdc[3441](info): commencing operation
krb5kdc: starting...
W20260620 06:57:43.386600 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:43 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:43.562835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34031
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:43.870106 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:43.870401 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:43.870576 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:43.877789 3454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:43.877899 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:43.877978 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:43.878057 3454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:43.878114 3454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:43.887712 3454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:43.889029 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:43.890352 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:43.901285 3459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:43.901257 3460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:43.901553 3462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:43.902024 3454 server_base.cc:1034] running on GCE node
I20260620 06:57:43.902787 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:43.904072 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:43.905278 3454 hybrid_clock.cc:648] HybridClock initialized: now 1781938663905223 us; error 38 us; skew 500 ppm
I20260620 06:57:43.905644 3454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:43 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:43.908548 3454 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:43.909863 3454 webserver.cc:460] Webserver started at http://127.0.97.62:42513/ using document root <none> and password file <none>
I20260620 06:57:43.910499 3454 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:43.910624 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:43.910873 3454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:43.913683 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "42633283da9747ce9ab38787b03a6139"
format_stamp: "Formatted at 2026-06-20 06:57:43 on dist-test-slave-mz20"
server_key: "d7d5b06deda5396676fe8678eb1c0cce"
server_key_iv: "a5917ba6cc5ab1cc7dca37510dd5df69"
server_key_version: "encryptionkey@0"
I20260620 06:57:43.914361 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "42633283da9747ce9ab38787b03a6139"
format_stamp: "Formatted at 2026-06-20 06:57:43 on dist-test-slave-mz20"
server_key: "d7d5b06deda5396676fe8678eb1c0cce"
server_key_iv: "a5917ba6cc5ab1cc7dca37510dd5df69"
server_key_version: "encryptionkey@0"
I20260620 06:57:43.919262 3454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 06:57:43.922422 3469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:43.923800 3454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:57:43.923944 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "42633283da9747ce9ab38787b03a6139"
format_stamp: "Formatted at 2026-06-20 06:57:43 on dist-test-slave-mz20"
server_key: "d7d5b06deda5396676fe8678eb1c0cce"
server_key_iv: "a5917ba6cc5ab1cc7dca37510dd5df69"
server_key_version: "encryptionkey@0"
I20260620 06:57:43.924124 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:43.968827 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:43.972815 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:43.973057 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:43.989658 3454 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34031
I20260620 06:57:43.989675 3520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34031 every 8 connection(s)
I20260620 06:57:43.990981 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:43.994611 3521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:43.998944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3454
I20260620 06:57:43.999154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:43.999714 388 external_mini_cluster.cc:1428] Setting key fdff9a47c78f134c5cd4ac52c13626e4
I20260620 06:57:44.004631 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139: Bootstrap starting.
I20260620 06:57:44.008013 3521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:44.009218 3521 log.cc:824] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:44.011997 3521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139: No bootstrap required, opened a new log
Jun 20 06:57:44 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:44.020567 3521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } }
I20260620 06:57:44.020933 3521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:44.021034 3521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42633283da9747ce9ab38787b03a6139, State: Initialized, Role: FOLLOWER
I20260620 06:57:44.021397 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:44.001563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.97.62:34031)
0620 06:57:44.002353 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:44.002364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:44.002404 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:57:44.004020 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:44.004042 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:44.004417 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:44.004749 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:44.006283 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.007555 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.009243 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.009963 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.010096 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:44.014230 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:44.014239 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:44.014256 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:44.014294 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:44.017300 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:44.018084 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:44.018108 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:44.018114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:44.018270 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:44.018662 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:44.018666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:44.018668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:44.019216 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:44.019440 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:44.019864 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 06:57:44.020239 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":348,"threads_started":1}
I20260620 06:57:44.021628 3521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } }
I20260620 06:57:44.021855 3521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:44.021986 3521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:44.022153 3521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:44.024654 3521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } }
I20260620 06:57:44.025126 3521 leader_election.cc:302] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42633283da9747ce9ab38787b03a6139; no voters:
I20260620 06:57:44.025661 3521 leader_election.cc:288] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:44.025767 3526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:44.026347 3526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [term 1 LEADER]: Becoming Leader. State: Replica: 42633283da9747ce9ab38787b03a6139, State: Running, Role: LEADER
I20260620 06:57:44.026906 3526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } }
I20260620 06:57:44.027552 3521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:44.029805 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42633283da9747ce9ab38787b03a6139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } } }
I20260620 06:57:44.030153 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:44.030879 3528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42633283da9747ce9ab38787b03a6139. Latest consensus state: current_term: 1 leader_uuid: "42633283da9747ce9ab38787b03a6139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42633283da9747ce9ab38787b03a6139" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34031 } } }
I20260620 06:57:44.031401 3528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:44.035701 3534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:44.040096 3534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:44.048390 3534 catalog_manager.cc:1347] Generated new cluster ID: 7580f10febd548e9a8a8f1f7e56c78da
I20260620 06:57:44.048502 3534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:44.070485 3534 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:44.072381 3534 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:44.079535 3534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42633283da9747ce9ab38787b03a6139: Generated new TSK 0
I20260620 06:57:44.080551 3534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:44.134595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:44.445905 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:44.446187 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:44.446318 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:44.453467 3547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:44.453576 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:44.453725 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:44.463248 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:44.464514 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:44.465772 3547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:44.477877 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:44.478000 3555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:44.477906 3553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:44.478859 3547 server_base.cc:1034] running on GCE node
I20260620 06:57:44.479472 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:44.480409 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:44.481597 3547 hybrid_clock.cc:648] HybridClock initialized: now 1781938664481538 us; error 60 us; skew 500 ppm
I20260620 06:57:44.481894 3547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:44 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:44.484848 3547 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:44.486305 3547 webserver.cc:460] Webserver started at http://127.0.97.1:38065/ using document root <none> and password file <none>
I20260620 06:57:44.487016 3547 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:44.487128 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:44.487397 3547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:44.490160 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "59b2ec8c3a9447cb850f63fdb2beef53"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "1ec36b0a712e2767fb43bb4dced528cc"
server_key_iv: "7ac4320b05dc040f69c9769c6fd3775b"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.490923 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "59b2ec8c3a9447cb850f63fdb2beef53"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "1ec36b0a712e2767fb43bb4dced528cc"
server_key_iv: "7ac4320b05dc040f69c9769c6fd3775b"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.495819 3547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 06:57:44.498975 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:44.500248 3547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 06:57:44.500388 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "59b2ec8c3a9447cb850f63fdb2beef53"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "1ec36b0a712e2767fb43bb4dced528cc"
server_key_iv: "7ac4320b05dc040f69c9769c6fd3775b"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.500548 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:44.523753 3547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:44.527911 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:44.528172 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:44.529198 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:44.530517 3547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:44.530642 3547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:44.530762 3547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:44.530813 3547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:44.559865 3547 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36185
I20260620 06:57:44.559912 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36185 every 8 connection(s)
I20260620 06:57:44.561066 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:44.562999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3547
I20260620 06:57:44.563187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:44.563961 388 external_mini_cluster.cc:1428] Setting key 34e941205b040d4dd1699167e4ff02e6
Jun 20 06:57:44 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:44.580197 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:44.563552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41251 (local address 127.0.97.62:34031)
0620 06:57:44.563992 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:44.563998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:44.565026 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 06:57:44.566776 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:44.566781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:44.566857 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:44.566968 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:44.568306 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.569301 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.570790 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.571171 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.571277 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:44.574753 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:44.574758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:44.574761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:44.574791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:44.577038 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:44.577868 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:44.577871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:44.577874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:44.577974 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:44.578392 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:44.578394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:44.578396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:44.578749 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:44.578861 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:44.579775 (+ 914us) server_negotiation.cc:299] Negotiation successful
0620 06:57:44.579931 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260620 06:57:44.581215 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:44.564060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34031 (local address 127.0.97.1:41251)
0620 06:57:44.564876 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:44.564920 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:44.566493 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:44.567139 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:44.567146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:44.567607 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:44.568153 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:44.568176 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.569416 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.569419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:44.570652 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:44.570678 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:44.571368 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:44.571376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:44.571589 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:44.572178 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:44.572200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:44.574528 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:44.577175 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:44.577184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:44.577203 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:44.577768 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:44.578060 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:44.578064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:44.578068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:44.578300 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:44.578875 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:44.578883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:44.579339 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 06:57:44.579625 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 06:57:44.580123 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":290,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 06:57:44.582775 3675 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34031
I20260620 06:57:44.583056 3675 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:44.583746 3675 heartbeater.cc:511] Master 127.0.97.62:34031 requested a full tablet report, sending...
I20260620 06:57:44.585915 3486 ts_manager.cc:194] Registered new tserver with Master: 59b2ec8c3a9447cb850f63fdb2beef53 (127.0.97.1:36185)
I20260620 06:57:44.587534 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:44.603813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:44.911438 3681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:44.911718 3681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:44.911856 3681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:44.918807 3681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:44.918921 3681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:44.919076 3681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:44.928610 3681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:44.929956 3681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:44.931295 3681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:44.943410 3687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:44.943480 3686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:44.943445 3689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:44.943948 3681 server_base.cc:1034] running on GCE node
I20260620 06:57:44.944711 3681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:44.945570 3681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:44.946760 3681 hybrid_clock.cc:648] HybridClock initialized: now 1781938664946721 us; error 51 us; skew 500 ppm
I20260620 06:57:44.947042 3681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:44 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:44.949957 3681 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:44.951411 3681 webserver.cc:460] Webserver started at http://127.0.97.2:40333/ using document root <none> and password file <none>
I20260620 06:57:44.952059 3681 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:44.952171 3681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:44.952446 3681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:44.955255 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "da02637f154b4fb39803967d76f6fec5"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "7405aa38b2b7c65a50e52b4ebf5a4505"
server_key_iv: "cda198dc94714c74e2b8b8b953cf8132"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.955981 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "da02637f154b4fb39803967d76f6fec5"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "7405aa38b2b7c65a50e52b4ebf5a4505"
server_key_iv: "cda198dc94714c74e2b8b8b953cf8132"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.960817 3681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 06:57:44.964061 3696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:44.965309 3681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:57:44.965451 3681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "da02637f154b4fb39803967d76f6fec5"
format_stamp: "Formatted at 2026-06-20 06:57:44 on dist-test-slave-mz20"
server_key: "7405aa38b2b7c65a50e52b4ebf5a4505"
server_key_iv: "cda198dc94714c74e2b8b8b953cf8132"
server_key_version: "encryptionkey@0"
I20260620 06:57:44.965615 3681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:44.996765 3681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:45.000993 3681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:45.001264 3681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:45.002183 3681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:45.003526 3681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:45.003599 3681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:45.003759 3681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:45.003810 3681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:45.032707 3681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40549
I20260620 06:57:45.032747 3808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40549 every 8 connection(s)
I20260620 06:57:45.033923 3681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:45.039770 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3681
I20260620 06:57:45.039970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:45.040470 388 external_mini_cluster.cc:1428] Setting key 5e2f8012989dec707acf016495706f2f
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:45.055816 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.036336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37865 (local address 127.0.97.62:34031)
0620 06:57:45.036498 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.036503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.037635 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 06:57:45.039435 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.039439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.039512 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.039601 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.041550 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.042650 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.044864 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.045294 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.045385 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.049434 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.049441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.049445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.049484 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.052586 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.053298 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.053303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.053306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.053473 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.053952 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.053957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.053960 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.054272 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.054407 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.055237 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.055460 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:57:45.056743 3811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.036803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34031 (local address 127.0.97.2:37865)
0620 06:57:45.037471 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:45.037517 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:45.039150 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:45.039882 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:45.039894 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:45.040601 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:45.041351 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:45.041377 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.042811 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.042816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:45.044707 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:45.044722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.045506 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.045513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:45.045767 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.046435 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:45.046462 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:45.049169 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:45.052747 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:45.052755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:45.052768 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:45.053188 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:45.053607 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:45.053611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:45.053613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:45.053837 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:45.054411 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:45.054418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:45.054867 (+ 449us) client_negotiation.cc:769] Sending connection context
0620 06:57:45.055134 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 06:57:45.055546 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":267,"threads_started":1}
I20260620 06:57:45.058099 3809 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34031
I20260620 06:57:45.058385 3809 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:45.059126 3809 heartbeater.cc:511] Master 127.0.97.62:34031 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:45.060518 3486 ts_manager.cc:194] Registered new tserver with Master: da02637f154b4fb39803967d76f6fec5 (127.0.97.2:40549)
I20260620 06:57:45.061414 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:45.081941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:45.394759 3814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:45.395062 3814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:45.395203 3814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:45.402311 3814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:45.402446 3814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:45.402637 3814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:45.412469 3814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34031
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:45.413771 3814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:45.415125 3814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:45.427569 3819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:45.427629 3822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:45.427639 3820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:45.428340 3814 server_base.cc:1034] running on GCE node
I20260620 06:57:45.428980 3814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:45.429936 3814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:45.431135 3814 hybrid_clock.cc:648] HybridClock initialized: now 1781938665431084 us; error 42 us; skew 500 ppm
I20260620 06:57:45.431457 3814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:45.434717 3814 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:45.436116 3814 webserver.cc:460] Webserver started at http://127.0.97.3:44057/ using document root <none> and password file <none>
I20260620 06:57:45.436816 3814 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:45.436919 3814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:45.437196 3814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:45.440120 3814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "4a383c50fc1a45cf86b63f1e97a85a75"
format_stamp: "Formatted at 2026-06-20 06:57:45 on dist-test-slave-mz20"
server_key: "65ffb4e098337f3a81bfa78d54884a1b"
server_key_iv: "9d441f9aa34af135ecb728b88a2b151a"
server_key_version: "encryptionkey@0"
I20260620 06:57:45.440896 3814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4a383c50fc1a45cf86b63f1e97a85a75"
format_stamp: "Formatted at 2026-06-20 06:57:45 on dist-test-slave-mz20"
server_key: "65ffb4e098337f3a81bfa78d54884a1b"
server_key_iv: "9d441f9aa34af135ecb728b88a2b151a"
server_key_version: "encryptionkey@0"
I20260620 06:57:45.445856 3814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:45.449110 3829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:45.450449 3814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:45.450655 3814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "4a383c50fc1a45cf86b63f1e97a85a75"
format_stamp: "Formatted at 2026-06-20 06:57:45 on dist-test-slave-mz20"
server_key: "65ffb4e098337f3a81bfa78d54884a1b"
server_key_iv: "9d441f9aa34af135ecb728b88a2b151a"
server_key_version: "encryptionkey@0"
I20260620 06:57:45.450819 3814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:45.476419 3814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:45.480477 3814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:45.480731 3814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:45.481626 3814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:45.482941 3814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:45.483012 3814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:45.483124 3814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:45.483171 3814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:45.511768 3814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42339
I20260620 06:57:45.511858 3941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42339 every 8 connection(s)
I20260620 06:57:45.512991 3814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:45.514106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3814
I20260620 06:57:45.514298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:45.514858 388 external_mini_cluster.cc:1428] Setting key 4fd59ecab2195510ab958da77ea26031
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:45.531602 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.515605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51361 (local address 127.0.97.62:34031)
0620 06:57:45.515761 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.515765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.517146 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 06:57:45.518933 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.518937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.518999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.519076 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.520450 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.521167 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.522674 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.523067 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.523158 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.526221 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.526226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.526230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.526260 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.528809 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.529570 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.529575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.529578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.529715 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.530100 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.530103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.530105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.530435 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.530563 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.531201 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.531364 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:57:45.532612 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.516214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34031 (local address 127.0.97.3:51361)
0620 06:57:45.516992 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:45.517042 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:45.518694 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:45.519260 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:45.519267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:45.519736 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:45.520301 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:45.520319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.521277 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.521281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:45.522486 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:45.522497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.523238 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.523243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:45.523439 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.524027 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:45.524047 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:45.526023 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:45.528931 (+ 2908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:45.528939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:45.528955 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:45.529464 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:45.529816 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:45.529821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:45.529823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:45.529994 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:45.530568 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:45.530576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:45.530896 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 06:57:45.531125 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:57:45.531525 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260620 06:57:45.533989 3942 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34031
I20260620 06:57:45.534286 3942 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:45.535064 3942 heartbeater.cc:511] Master 127.0.97.62:34031 requested a full tablet report, sending...
I20260620 06:57:45.536428 3486 ts_manager.cc:194] Registered new tserver with Master: 4a383c50fc1a45cf86b63f1e97a85a75 (127.0.97.3:42339)
I20260620 06:57:45.537163 3486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51361
I20260620 06:57:45.541640 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:45.585902 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.573600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.0.97.62:34031)
0620 06:57:45.573855 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.573862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.574177 (+ 315us) server_negotiation.cc:407] Connection header received
0620 06:57:45.574305 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.574309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.574374 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.574458 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.575695 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.576402 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.577595 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.578024 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.578115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.581514 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.581518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.581521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.581559 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.583832 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.584343 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.584346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.584348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.584450 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.584739 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.584742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.584744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.585057 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.585219 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.585446 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.585599 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 06:57:45.586431 3486 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41248
I20260620 06:57:45.587616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3547
I20260620 06:57:45.590847 3675 heartbeater.cc:503] Master 127.0.97.62:34031 was elected leader, sending a full tablet report...
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:45.602324 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.588679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.0.97.1:36185)
0620 06:57:45.589260 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.589267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.589291 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:45.589347 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.589353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.589556 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.589676 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.590837 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.591978 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.593177 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.593717 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.593810 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.597334 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.597345 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.597360 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.597392 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.600053 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.600634 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.600640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.600643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.600796 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.601067 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.601071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.601073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.601388 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.601596 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.601805 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.601979 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":314,"threads_started":1}
W20260620 06:57:45.603551 3670 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55840
W20260620 06:57:45.604606 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:45.604773 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3547
I20260620 06:57:45.616786 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3681
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:45.632189 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.617644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.97.2:40549)
0620 06:57:45.618147 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.618155 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.618178 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:57:45.618238 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.618244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.618447 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.618612 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.619653 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.620648 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.621830 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.622512 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.622650 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.626200 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.626211 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.626225 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.626256 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.629143 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.629847 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.629855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.629857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.629974 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.630429 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.630434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.630436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.630957 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.631229 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.631554 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.631826 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":267,"threads_started":1}
W20260620 06:57:45.633466 3804 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35290
W20260620 06:57:45.634596 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3681: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:45.634800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3681
I20260620 06:57:45.646711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3814
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938665, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:45.662006 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:45.647577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.97.3:42339)
0620 06:57:45.648229 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:45.648236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:45.648264 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:57:45.648329 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:45.648335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:45.648537 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:45.648674 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:45.649906 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.651006 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.652187 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:45.652731 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:45.652840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:45.657058 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:45.657069 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:45.657083 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:45.657123 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:45.659727 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.660344 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.660350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.660353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.660457 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:45.660726 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:45.660729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:45.660731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:45.661031 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:45.661240 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:45.661473 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 06:57:45.661661 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":360,"threads_started":1}
W20260620 06:57:45.663338 3937 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37900
W20260620 06:57:45.664387 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3814: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 06:57:45.664517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3814
I20260620 06:57:45.676540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3454
I20260620 06:57:45.677685 3516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:45.777989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3454
2026-06-20T06:57:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3391 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3959](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3959](info): set up 2 sockets
Jun 20 06:57:45 dist-test-slave-mz20 krb5kdc[3959](info): commencing operation
krb5kdc: starting...
W20260620 06:57:46.767066 388 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:46 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:46.964324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39371
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:47.319533 3972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:47.319933 3972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:47.320058 3972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:47.327982 3972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:47.328123 3972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:47.328183 3972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:47.328277 3972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:47.328328 3972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:47.339334 3972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:47.341101 3972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:47.342795 3972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:47.355698 3977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:47.355634 3980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:47.355655 3978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:47.356650 3972 server_base.cc:1034] running on GCE node
I20260620 06:57:47.357507 3972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:47.359148 3972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:47.360363 3972 hybrid_clock.cc:648] HybridClock initialized: now 1781938667360316 us; error 39 us; skew 500 ppm
I20260620 06:57:47.360728 3972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:47 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:47.366922 3972 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:47.368698 3972 webserver.cc:460] Webserver started at http://127.0.97.62:43177/ using document root <none> and password file <none>
I20260620 06:57:47.369700 3972 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:47.369850 3972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:47.370280 3972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:47.373683 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "59aa8dce607d4b34b768722f31b45e4c"
format_stamp: "Formatted at 2026-06-20 06:57:47 on dist-test-slave-mz20"
server_key: "5741e4c7d2e47c75d96959e5ba2cbc3d"
server_key_iv: "c13d513cb59d26c88141c6d43abf739f"
server_key_version: "encryptionkey@0"
I20260620 06:57:47.374470 3972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "59aa8dce607d4b34b768722f31b45e4c"
format_stamp: "Formatted at 2026-06-20 06:57:47 on dist-test-slave-mz20"
server_key: "5741e4c7d2e47c75d96959e5ba2cbc3d"
server_key_iv: "c13d513cb59d26c88141c6d43abf739f"
server_key_version: "encryptionkey@0"
I20260620 06:57:47.380234 3972 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:47.384068 3987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:47.385583 3972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:47.385816 3972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "59aa8dce607d4b34b768722f31b45e4c"
format_stamp: "Formatted at 2026-06-20 06:57:47 on dist-test-slave-mz20"
server_key: "5741e4c7d2e47c75d96959e5ba2cbc3d"
server_key_iv: "c13d513cb59d26c88141c6d43abf739f"
server_key_version: "encryptionkey@0"
I20260620 06:57:47.386055 3972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:47.429916 3972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:47.435029 3972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:47.435344 3972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:47.454502 3972 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39371
I20260620 06:57:47.454560 4038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39371 every 8 connection(s)
I20260620 06:57:47.456113 3972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:47.460649 4039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:47.464349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 3972
I20260620 06:57:47.464568 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:47.465453 388 external_mini_cluster.cc:1428] Setting key 7d6bceedf8ce565ff34373cf90069617
I20260620 06:57:47.479624 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c: Bootstrap starting.
I20260620 06:57:47.484511 4039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:47.485925 4039 log.cc:824] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:47.490176 4039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c: No bootstrap required, opened a new log
Jun 20 06:57:47 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:47.501793 4039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } }
I20260620 06:57:47.502345 4039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:47.502458 4039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59aa8dce607d4b34b768722f31b45e4c, State: Initialized, Role: FOLLOWER
I20260620 06:57:47.502842 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:47.468107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.0.97.62:39371)
0620 06:57:47.474983 (+ 6876us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:47.475007 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:47.475082 (+ 75us) server_negotiation.cc:407] Connection header received
0620 06:57:47.477795 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:47.477838 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:47.478584 (+ 746us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:47.479166 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:47.481910 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:47.484045 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:47.485951 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:47.487061 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:47.487267 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:47.492565 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:47.492573 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:47.492594 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:47.492640 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:47.496613 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:47.497341 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:47.497350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:47.497365 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:47.497613 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:47.497990 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:47.497995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:47.497999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:47.498892 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:47.499244 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:47.500569 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0620 06:57:47.501121 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6437,"thread_start_us":6152,"threads_started":1}
I20260620 06:57:47.503320 4039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } }
I20260620 06:57:47.503572 4039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:47.503717 4039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:47.503913 4039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:47.507200 4039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } }
I20260620 06:57:47.507778 4039 leader_election.cc:302] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59aa8dce607d4b34b768722f31b45e4c; no voters:
I20260620 06:57:47.508411 4039 leader_election.cc:288] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:47.508536 4044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:47.509714 4044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [term 1 LEADER]: Becoming Leader. State: Replica: 59aa8dce607d4b34b768722f31b45e4c, State: Running, Role: LEADER
I20260620 06:57:47.510335 4044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } }
I20260620 06:57:47.510854 4039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:47.513180 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59aa8dce607d4b34b768722f31b45e4c. Latest consensus state: current_term: 1 leader_uuid: "59aa8dce607d4b34b768722f31b45e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } } }
I20260620 06:57:47.513612 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:47.514137 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59aa8dce607d4b34b768722f31b45e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59aa8dce607d4b34b768722f31b45e4c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39371 } } }
I20260620 06:57:47.514412 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:47.518482 4050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:47.523967 4050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:47.534416 4050 catalog_manager.cc:1347] Generated new cluster ID: 90da51b97ae749248dfe59f6996d549b
I20260620 06:57:47.534602 4050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:47.553753 4050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:47.555473 4050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:47.563933 4050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59aa8dce607d4b34b768722f31b45e4c: Generated new TSK 0
I20260620 06:57:47.565232 4050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:47.633903 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:48.037542 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:48.037918 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:48.038053 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:48.045866 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:48.045995 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:48.046155 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:48.056516 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:48.058148 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:48.059734 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:48.074347 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:48.074492 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:48.074504 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:48.075273 4066 server_base.cc:1034] running on GCE node
I20260620 06:57:48.075974 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:48.077128 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:48.078387 4066 hybrid_clock.cc:648] HybridClock initialized: now 1781938668078323 us; error 52 us; skew 500 ppm
I20260620 06:57:48.078783 4066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:48 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:48.082235 4066 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:48.083957 4066 webserver.cc:460] Webserver started at http://127.0.97.1:40733/ using document root <none> and password file <none>
I20260620 06:57:48.084753 4066 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:48.084872 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:48.085196 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:48.088559 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "29768ef7310a47bfa4db8b70a7857f4e"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "3492b80b9724f131c0d0b991bd4d0656"
server_key_iv: "248277552fc0f5f63b4dc2f7dc992fa8"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.089408 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "29768ef7310a47bfa4db8b70a7857f4e"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "3492b80b9724f131c0d0b991bd4d0656"
server_key_iv: "248277552fc0f5f63b4dc2f7dc992fa8"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.095162 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 06:57:48.098989 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.100536 4066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:48.100682 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "29768ef7310a47bfa4db8b70a7857f4e"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "3492b80b9724f131c0d0b991bd4d0656"
server_key_iv: "248277552fc0f5f63b4dc2f7dc992fa8"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.100878 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:48.119235 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:48.123889 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:48.124179 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:48.125263 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:48.126775 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:48.126863 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.126996 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:48.127051 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.160975 4066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44445
I20260620 06:57:48.161010 4193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44445 every 8 connection(s)
I20260620 06:57:48.162431 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:48.168960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4066
I20260620 06:57:48.169183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:48.170058 388 external_mini_cluster.cc:1428] Setting key 1eb89221bd0edb1beafa93bb97672c7c
Jun 20 06:57:48 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:48.186508 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:48.165326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55523 (local address 127.0.97.62:39371)
0620 06:57:48.165840 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:48.165847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:48.167037 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 06:57:48.169177 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:48.169182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:48.169268 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:48.169384 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:48.171562 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.172597 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.175074 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.175502 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.175608 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:48.180050 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:48.180057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:48.180062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:48.180103 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:48.183304 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:48.184039 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:48.184045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:48.184048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:48.184187 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:48.184659 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:48.184664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:48.184667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:48.185020 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:48.185167 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:48.185917 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 06:57:48.186144 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260620 06:57:48.187551 4197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:48.165975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39371 (local address 127.0.97.1:55523)
0620 06:57:48.166854 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:48.166899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:48.168840 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:48.169690 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:48.169703 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:48.170492 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:48.171388 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:48.171411 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.172766 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.172771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:48.174935 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:48.174950 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.175739 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.175745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:48.176050 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:48.176848 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:48.176883 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:48.179771 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:48.183444 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:48.183452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:48.183469 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:48.183915 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:48.184355 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:48.184359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:48.184362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:48.184558 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:48.185122 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:48.185129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:48.185534 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 06:57:48.185785 (+ 251us) client_negotiation.cc:240] Negotiation successful
0620 06:57:48.186228 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":342,"threads_started":1}
I20260620 06:57:48.189440 4194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39371
I20260620 06:57:48.189802 4194 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:48.190738 4194 heartbeater.cc:511] Master 127.0.97.62:39371 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:48.193257 4004 ts_manager.cc:194] Registered new tserver with Master: 29768ef7310a47bfa4db8b70a7857f4e (127.0.97.1:44445)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 06:57:48.195039 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:48.219579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:48.575492 4200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:48.575800 4200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:48.575915 4200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:48.583132 4200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:48.583252 4200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:48.583405 4200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:48.593256 4200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:48.594652 4200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:48.596056 4200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:48.608942 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:48.608878 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:48.608916 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:48.609615 4200 server_base.cc:1034] running on GCE node
I20260620 06:57:48.610193 4200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:48.611168 4200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:48.612373 4200 hybrid_clock.cc:648] HybridClock initialized: now 1781938668612334 us; error 36 us; skew 500 ppm
I20260620 06:57:48.612694 4200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:48 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:48.615823 4200 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:48.617218 4200 webserver.cc:460] Webserver started at http://127.0.97.2:41057/ using document root <none> and password file <none>
I20260620 06:57:48.617913 4200 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:48.618026 4200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:48.618366 4200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:48.621455 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f9afad59c714e1a9ce4dd7467c64a95"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "d07ee4c63fa7c2bbfb82763585941031"
server_key_iv: "5a1f25d89998cdaa9ea4ea56a14700eb"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.622219 4200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f9afad59c714e1a9ce4dd7467c64a95"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "d07ee4c63fa7c2bbfb82763585941031"
server_key_iv: "5a1f25d89998cdaa9ea4ea56a14700eb"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.627568 4200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 06:57:48.631173 4215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.632594 4200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:48.632773 4200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "4f9afad59c714e1a9ce4dd7467c64a95"
format_stamp: "Formatted at 2026-06-20 06:57:48 on dist-test-slave-mz20"
server_key: "d07ee4c63fa7c2bbfb82763585941031"
server_key_iv: "5a1f25d89998cdaa9ea4ea56a14700eb"
server_key_version: "encryptionkey@0"
I20260620 06:57:48.632961 4200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:48.646469 4200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:48.650648 4200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:48.650928 4200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:48.651903 4200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:48.653220 4200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:48.653296 4200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.653407 4200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:48.653461 4200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:48.683235 4200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42463
I20260620 06:57:48.683300 4327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42463 every 8 connection(s)
I20260620 06:57:48.684485 4200 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:48.689911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4200
I20260620 06:57:48.690099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:48.690709 388 external_mini_cluster.cc:1428] Setting key fa54ceec158de891d1a85c1fafbe3a1b
Jun 20 06:57:48 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:48.706799 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:48.687167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32965 (local address 127.0.97.62:39371)
0620 06:57:48.687367 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:48.687374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:48.688709 (+ 1335us) server_negotiation.cc:407] Connection header received
0620 06:57:48.691037 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:48.691052 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:48.691172 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:48.691271 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:48.693132 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.693856 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.696013 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.696423 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.696514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:48.700482 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:48.700489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:48.700495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:48.700533 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:48.703497 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:48.704208 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:48.704212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:48.704216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:48.704364 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:48.704780 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:48.704784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:48.704787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:48.705127 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:48.705270 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:48.706165 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 06:57:48.706407 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:57:48.707741 4330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:48.687725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39371 (local address 127.0.97.2:32965)
0620 06:57:48.688530 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:48.688587 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:48.690673 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:48.691540 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:48.691550 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:48.692214 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:48.692957 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:48.692982 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.694015 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.694021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:48.695886 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:48.695899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:48.696631 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:48.696637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:48.696907 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:48.697551 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:48.697577 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:48.700226 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:48.703642 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:48.703650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:48.703664 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:48.704084 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:48.704466 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:48.704469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:48.704471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:48.704671 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:48.705273 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:48.705280 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:48.705724 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 06:57:48.706019 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 06:57:48.706566 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":347,"threads_started":1}
I20260620 06:57:48.709187 4328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39371
I20260620 06:57:48.709534 4328 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:48.710443 4328 heartbeater.cc:511] Master 127.0.97.62:39371 requested a full tablet report, sending...
I20260620 06:57:48.712121 4004 ts_manager.cc:194] Registered new tserver with Master: 4f9afad59c714e1a9ce4dd7467c64a95 (127.0.97.2:42463)
I20260620 06:57:48.712970 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:48.737500 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:49.060412 4333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:49.060686 4333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:49.060781 4333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:49.067833 4333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:49.067945 4333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:49.068058 4333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:49.077782 4333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:49.079084 4333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:49.080360 4333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:49.092659 4338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:49.092615 4339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:49.092687 4341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:49.093696 4333 server_base.cc:1034] running on GCE node
I20260620 06:57:49.094256 4333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:49.095254 4333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:49.096453 4333 hybrid_clock.cc:648] HybridClock initialized: now 1781938669096393 us; error 50 us; skew 500 ppm
I20260620 06:57:49.096746 4333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:49.099583 4333 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:49.101004 4333 webserver.cc:460] Webserver started at http://127.0.97.3:37585/ using document root <none> and password file <none>
I20260620 06:57:49.101643 4333 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:49.101753 4333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:49.102020 4333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:49.104782 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "669e32f331e949cb9eba6c2d8b3bb823"
format_stamp: "Formatted at 2026-06-20 06:57:49 on dist-test-slave-mz20"
server_key: "d37ceea2378430ae2983b5c7bd435490"
server_key_iv: "7a0f28760e45e646253dcfc18dac2a46"
server_key_version: "encryptionkey@0"
I20260620 06:57:49.105540 4333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "669e32f331e949cb9eba6c2d8b3bb823"
format_stamp: "Formatted at 2026-06-20 06:57:49 on dist-test-slave-mz20"
server_key: "d37ceea2378430ae2983b5c7bd435490"
server_key_iv: "7a0f28760e45e646253dcfc18dac2a46"
server_key_version: "encryptionkey@0"
I20260620 06:57:49.110308 4333 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 06:57:49.113520 4348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:49.114880 4333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:57:49.115020 4333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "669e32f331e949cb9eba6c2d8b3bb823"
format_stamp: "Formatted at 2026-06-20 06:57:49 on dist-test-slave-mz20"
server_key: "d37ceea2378430ae2983b5c7bd435490"
server_key_iv: "7a0f28760e45e646253dcfc18dac2a46"
server_key_version: "encryptionkey@0"
I20260620 06:57:49.115195 4333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:49.127619 4333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:49.131850 4333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:49.132108 4333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:49.133023 4333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:49.134337 4333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:49.134406 4333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:49.134488 4333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:49.134532 4333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:49.165287 4333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42539
I20260620 06:57:49.165318 4460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42539 every 8 connection(s)
I20260620 06:57:49.166450 4333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:49.174045 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4333
I20260620 06:57:49.174185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:49.174619 388 external_mini_cluster.cc:1428] Setting key f956c4881dae1a8403a99fed97697eba
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:49.184963 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.168805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37685 (local address 127.0.97.62:39371)
0620 06:57:49.168964 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.168968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.170130 (+ 1162us) server_negotiation.cc:407] Connection header received
0620 06:57:49.171902 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.171907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.171981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.172065 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.173461 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.174239 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.175947 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.176331 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.176439 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.179594 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.179599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.179602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.179633 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.182006 (+ 2373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.182770 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.182774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.182776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.182897 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.183302 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.183305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.183307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.183560 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.183686 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.184393 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.184612 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:57:49.185863 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.169295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39371 (local address 127.0.97.3:37685)
0620 06:57:49.169981 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:49.170019 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:49.171644 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:49.172299 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:49.172306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:49.172766 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:49.173299 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:49.173317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.174373 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.174377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:49.175822 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:49.175836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.176544 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.176551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:49.176768 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.177362 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:49.177384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:49.179404 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:49.182154 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:49.182163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:49.182180 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:49.182645 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:49.183002 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:49.183007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:49.183011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:49.183189 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:49.183680 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:49.183688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:49.184020 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 06:57:49.184255 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 06:57:49.184769 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
I20260620 06:57:49.187276 4461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39371
I20260620 06:57:49.187523 4461 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:49.188185 4461 heartbeater.cc:511] Master 127.0.97.62:39371 requested a full tablet report, sending...
I20260620 06:57:49.189591 4004 ts_manager.cc:194] Registered new tserver with Master: 669e32f331e949cb9eba6c2d8b3bb823 (127.0.97.3:42539)
I20260620 06:57:49.190325 4004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37685
I20260620 06:57:49.198848 4194 heartbeater.cc:503] Master 127.0.97.62:39371 was elected leader, sending a full tablet report...
I20260620 06:57:49.201572 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:49.246433 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.231963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.0.97.1:44445)
0620 06:57:49.232595 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.232602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.232629 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:57:49.232688 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.232695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.232919 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.233049 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.234362 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.235390 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.236614 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.237290 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.237400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.241115 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.241171 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.241188 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.241224 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.243944 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.244545 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.244553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.244555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.244692 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.244984 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.244987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.244989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.245355 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.245576 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.245824 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.246065 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":331,"threads_started":1}
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:49.262818 4196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.250465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.97.62:39371)
0620 06:57:49.250707 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.250714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.250851 (+ 137us) server_negotiation.cc:407] Connection header received
0620 06:57:49.251074 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.251079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.251160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.251240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.252287 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.253012 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.254163 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.254657 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.254762 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.258411 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.258416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.258419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.258453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.260697 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.261283 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.261286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.261288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.261403 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.261661 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.261664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.261666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.261952 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.262115 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.262332 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.262492 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260620 06:57:49.263327 4004 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45894
I20260620 06:57:49.264281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4066
I20260620 06:57:49.276201 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.265102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.97.1:44445)
0620 06:57:49.265304 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.265310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.265451 (+ 141us) server_negotiation.cc:407] Connection header received
0620 06:57:49.265593 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.265597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.265656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.265731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.266979 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.267675 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.268837 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.269282 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.269372 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.271636 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.271641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.271644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.271676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.274074 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.274719 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.274723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.274725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.274840 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.275127 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.275130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.275132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.275360 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.275530 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.275745 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.275905 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:57:49.276837 4189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:49.357642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4066
I20260620 06:57:49.370134 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4200
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:49.385093 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.370984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.97.2:42463)
0620 06:57:49.371581 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.371588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.371614 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:57:49.371675 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.371681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.371893 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.372027 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.373137 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.374217 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.375536 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.376086 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.376194 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.379989 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.380000 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.380015 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.380047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.382786 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.383382 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.383389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.383391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.383509 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.383779 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.383783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.383785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.384108 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.384339 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.384548 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.384744 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":347,"threads_started":1}
I20260620 06:57:49.386325 4323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:49.467434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4200
I20260620 06:57:49.483528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4333
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[3959](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:49.499150 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:49.484392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.0.97.3:42539)
0620 06:57:49.484957 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:49.484964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:49.484992 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:57:49.485047 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:49.485052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:49.485257 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:49.485399 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:49.486684 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.487752 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.489006 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:49.489682 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:49.489822 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:49.493976 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:49.493989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:49.494003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:49.494038 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:49.496891 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.497495 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.497501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.497503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.497609 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:49.497872 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:49.497875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:49.497877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:49.498175 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:49.498415 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:49.498610 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:57:49.498804 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":243,"threads_started":1}
I20260620 06:57:49.500360 4456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:49.580713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4333
I20260620 06:57:49.593214 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 3972
I20260620 06:57:49.594345 4034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:49.694347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 3972
2026-06-20T06:57:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3917 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[4480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[4480](info): set up 2 sockets
Jun 20 06:57:49 dist-test-slave-mz20 krb5kdc[4480](info): commencing operation
krb5kdc: starting...
W20260620 06:57:50.699155 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:57:50 dist-test-slave-mz20 krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:57:50.877887 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43883
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:51.182202 4493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:51.182550 4493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:51.182680 4493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:51.190114 4493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:51.190223 4493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:51.190275 4493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:51.190352 4493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:51.190424 4493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:51.200306 4493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:51.201596 4493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:51.202943 4493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:51.214147 4501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:51.214241 4499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:51.214177 4498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:51.215083 4493 server_base.cc:1034] running on GCE node
I20260620 06:57:51.215883 4493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:51.217206 4493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:51.218405 4493 hybrid_clock.cc:648] HybridClock initialized: now 1781938671218350 us; error 48 us; skew 500 ppm
I20260620 06:57:51.218792 4493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:51 dist-test-slave-mz20 krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:51.221861 4493 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:57:51.223052 4493 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:51.223141 4493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:51.223450 4493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:51.226365 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "31eb6bf4f77e4b29b32c70d878530c54"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "b6e1903aea3a72e5cc34a28bc6fee28e"
server_key_iv: "f12004ec4ac35c64183029a2fdd20e5b"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.227180 4493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "31eb6bf4f77e4b29b32c70d878530c54"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "b6e1903aea3a72e5cc34a28bc6fee28e"
server_key_iv: "f12004ec4ac35c64183029a2fdd20e5b"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.232069 4493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:51.235606 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:51.237048 4493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:57:51.237231 4493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "31eb6bf4f77e4b29b32c70d878530c54"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "b6e1903aea3a72e5cc34a28bc6fee28e"
server_key_iv: "f12004ec4ac35c64183029a2fdd20e5b"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.237440 4493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:51.256057 4493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:51.259922 4493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:51.260154 4493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:51.276989 4493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43883
I20260620 06:57:51.277007 4558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43883 every 8 connection(s)
I20260620 06:57:51.277825 4493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:51.281689 4559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:51.284319 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4493
I20260620 06:57:51.284509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:51.285109 388 external_mini_cluster.cc:1428] Setting key 9ccbba10c01058cfe61e88a1ecd4c8a4
I20260620 06:57:51.294377 4559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: Bootstrap starting.
I20260620 06:57:51.297871 4559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:51.298858 4559 log.cc:824] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:57:51 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:51.301630 4559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: No bootstrap required, opened a new log
I20260620 06:57:51.308923 4562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:51.287080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.97.62:43883)
0620 06:57:51.287926 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:51.287951 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:51.288001 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:57:51.290186 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:51.290215 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:51.290805 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:51.291278 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:51.292647 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.294520 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.295829 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.296834 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.297021 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:51.300683 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:51.300694 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:51.300713 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:51.300754 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:51.304357 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:51.305026 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:51.305035 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:51.305044 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:51.305243 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:51.305656 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:51.305661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:51.305665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:51.306357 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:51.306675 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:51.307098 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 06:57:51.307542 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":376,"threads_started":1}
I20260620 06:57:51.311228 4559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } }
I20260620 06:57:51.311627 4559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:51.311734 4559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31eb6bf4f77e4b29b32c70d878530c54, State: Initialized, Role: FOLLOWER
I20260620 06:57:51.312354 4559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } }
I20260620 06:57:51.312569 4559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:51.312660 4559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:51.312791 4559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:51.314973 4559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } }
I20260620 06:57:51.315459 4559 leader_election.cc:302] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31eb6bf4f77e4b29b32c70d878530c54; no voters:
I20260620 06:57:51.315963 4559 leader_election.cc:288] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:51.316092 4564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:51.317179 4564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [term 1 LEADER]: Becoming Leader. State: Replica: 31eb6bf4f77e4b29b32c70d878530c54, State: Running, Role: LEADER
I20260620 06:57:51.317755 4564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } }
I20260620 06:57:51.318223 4559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:51.320420 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31eb6bf4f77e4b29b32c70d878530c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } } }
I20260620 06:57:51.320577 4566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31eb6bf4f77e4b29b32c70d878530c54. Latest consensus state: current_term: 1 leader_uuid: "31eb6bf4f77e4b29b32c70d878530c54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31eb6bf4f77e4b29b32c70d878530c54" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43883 } } }
I20260620 06:57:51.320797 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:51.320827 4566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:51.321693 4570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:51.328490 4570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:51.338025 4570 catalog_manager.cc:1347] Generated new cluster ID: 5021260a00314b63b9fc6a631cad728f
I20260620 06:57:51.338152 4570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:51.353623 4570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:51.354936 4570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:51.359699 4570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: Generated new TSK 0
I20260620 06:57:51.360435 4570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:57:51.412026 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:51.735123 4585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:51.735409 4585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:51.735503 4585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:51.742878 4585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:51.742996 4585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:51.743125 4585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:51.752684 4585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:51.753965 4585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:51.755182 4585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:51.767006 4593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:51.767006 4591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:51.767168 4590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:51.767578 4585 server_base.cc:1034] running on GCE node
I20260620 06:57:51.768157 4585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:51.769058 4585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:51.770253 4585 hybrid_clock.cc:648] HybridClock initialized: now 1781938671770208 us; error 35 us; skew 500 ppm
I20260620 06:57:51.770597 4585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:51 dist-test-slave-mz20 krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:51.773681 4585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:57:51.774816 4585 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:51.774917 4585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:51.775208 4585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:51.778023 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "7e1ac3fa03174df69d4ef508d78a38b2"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "23eff9d4e368d44436eaec24ccb6b323"
server_key_iv: "21aadd5ff95a04eb9a35dd403e58e34e"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.778860 4585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7e1ac3fa03174df69d4ef508d78a38b2"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "23eff9d4e368d44436eaec24ccb6b323"
server_key_iv: "21aadd5ff95a04eb9a35dd403e58e34e"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.783878 4585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:51.787607 4599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:51.789280 4585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:51.789417 4585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "7e1ac3fa03174df69d4ef508d78a38b2"
format_stamp: "Formatted at 2026-06-20 06:57:51 on dist-test-slave-mz20"
server_key: "23eff9d4e368d44436eaec24ccb6b323"
server_key_iv: "21aadd5ff95a04eb9a35dd403e58e34e"
server_key_version: "encryptionkey@0"
I20260620 06:57:51.789592 4585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:51.812126 4585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:51.816073 4585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:51.816293 4585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:51.817169 4585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:51.818400 4585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:51.818487 4585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:51.818604 4585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:51.818663 4585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:51.847920 4585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42967
I20260620 06:57:51.847971 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42967 every 8 connection(s)
I20260620 06:57:51.848723 4585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:51.852035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4585
I20260620 06:57:51.852227 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:51.852959 388 external_mini_cluster.cc:1428] Setting key 09c5d3fec942fe6e1cc0c60ee69c9909
Jun 20 06:57:51 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:51.868633 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:51.851031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42005 (local address 127.0.97.62:43883)
0620 06:57:51.851436 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:51.851441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:51.852564 (+ 1123us) server_negotiation.cc:407] Connection header received
0620 06:57:51.854488 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:51.854493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:51.854595 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:51.854717 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:51.856171 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.857172 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.858709 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.859259 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.859378 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:51.863207 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:51.863212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:51.863216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:51.863247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:51.865511 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:51.866332 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:51.866335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:51.866338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:51.866452 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:51.866955 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:51.866958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:51.866960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:51.867221 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:51.867322 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:51.868213 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 06:57:51.868370 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":211,"threads_started":1}
I20260620 06:57:51.870148 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:51.851597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43883 (local address 127.0.97.1:42005)
0620 06:57:51.852411 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:51.852450 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:51.854223 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:51.854943 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:51.854952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:51.855456 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:51.855994 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:51.856009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.857320 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.857324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:51.858578 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:51.858593 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:51.859482 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:51.859487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:51.859672 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:51.860374 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:51.860402 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:51.862971 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:51.865651 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:51.865660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:51.865677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:51.866227 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:51.866588 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:51.866593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:51.866598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:51.866841 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:51.867352 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:51.867361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:51.867832 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 06:57:51.868154 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 06:57:51.868715 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:57:51.871706 4712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43883
I20260620 06:57:51.871992 4712 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:51.872701 4712 heartbeater.cc:511] Master 127.0.97.62:43883 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 06:57:51.874939 4524 ts_manager.cc:194] Registered new tserver with Master: 7e1ac3fa03174df69d4ef508d78a38b2 (127.0.97.1:42967)
I20260620 06:57:51.876745 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:57:51.895550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:52.199887 4718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:52.200188 4718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:52.200318 4718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:52.207453 4718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:52.207583 4718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:52.207752 4718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:52.217509 4718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:52.218852 4718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:52.220118 4718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:52.232784 4726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:52.232751 4723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:52.232822 4724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:52.239043 4718 server_base.cc:1034] running on GCE node
I20260620 06:57:52.239764 4718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:52.240856 4718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:52.242093 4718 hybrid_clock.cc:648] HybridClock initialized: now 1781938672242028 us; error 52 us; skew 500 ppm
I20260620 06:57:52.242417 4718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:52 dist-test-slave-mz20 krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:52.245671 4718 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:57:52.246876 4718 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:52.246977 4718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:52.247269 4718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:52.250381 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "8df3869c07ec420a800d19b5db8ce9c9"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "546080df4cfe98b216e715b0f7b9e448"
server_key_iv: "6d3ba066bba65a75f23c2d3919775dda"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.251360 4718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8df3869c07ec420a800d19b5db8ce9c9"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "546080df4cfe98b216e715b0f7b9e448"
server_key_iv: "6d3ba066bba65a75f23c2d3919775dda"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.256382 4718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:57:52.259806 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.261189 4718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:52.261346 4718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "8df3869c07ec420a800d19b5db8ce9c9"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "546080df4cfe98b216e715b0f7b9e448"
server_key_iv: "6d3ba066bba65a75f23c2d3919775dda"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.261533 4718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:52.283746 4718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:52.288889 4718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:52.289182 4718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:52.290378 4718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:52.292045 4718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:52.292148 4718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.292263 4718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:52.292341 4718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.320144 4718 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41541
I20260620 06:57:52.320180 4844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41541 every 8 connection(s)
I20260620 06:57:52.320976 4718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:52.322365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4718
I20260620 06:57:52.322499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:52.322971 388 external_mini_cluster.cc:1428] Setting key 7e4aaaf566d4b2983ccd3f9add93ce62
Jun 20 06:57:52 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:57:52.342234 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.323527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43125 (local address 127.0.97.62:43883)
0620 06:57:52.323735 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.323742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.325088 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 06:57:52.327555 (+ 2467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.327560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.327633 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.327758 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:52.329361 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.330117 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.331993 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.332443 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.332542 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.336363 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:52.336368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:52.336372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:52.336404 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:52.338926 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.339767 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.339771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.339773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.339891 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.340379 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.340384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.340386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.340659 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:52.340833 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.341657 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.341894 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 06:57:52.343261 4847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.324106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43883 (local address 127.0.97.2:43125)
0620 06:57:52.324942 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:52.324979 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:52.327259 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:52.328011 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:52.328020 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:52.328617 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:52.329188 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.329207 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.330263 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.330267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.331844 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.331859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.332668 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.332674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.332923 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.333608 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:52.333633 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:52.336146 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:52.339055 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:52.339064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:52.339083 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:52.339647 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:52.339995 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:52.339999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:52.340003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:52.340224 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:52.340875 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:52.340883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:52.341250 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 06:57:52.341487 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 06:57:52.342015 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 06:57:52.344882 4845 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43883
I20260620 06:57:52.345189 4845 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:52.345816 4845 heartbeater.cc:511] Master 127.0.97.62:43883 requested a full tablet report, sending...
I20260620 06:57:52.347281 4524 ts_manager.cc:194] Registered new tserver with Master: 8df3869c07ec420a800d19b5db8ce9c9 (127.0.97.2:41541)
I20260620 06:57:52.348228 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:57:52.363669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:57:52.692392 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:52.692736 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:52.692881 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:52.700376 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:52.700505 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:52.700660 4850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:52.710664 4850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:52.712246 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:52.713779 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:52.728140 4856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:52.728111 4858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:52.728080 4855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:52.729463 4850 server_base.cc:1034] running on GCE node
I20260620 06:57:52.730178 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:52.731546 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:52.732784 4850 hybrid_clock.cc:648] HybridClock initialized: now 1781938672732722 us; error 52 us; skew 500 ppm
I20260620 06:57:52.733191 4850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:57:52 dist-test-slave-mz20 krb5kdc[4480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:57:52.740170 4850 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:57:52.741560 4850 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:52.741654 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:52.742017 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:52.745571 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "25237e76de584d0889f247e65bba9d88"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "218cb4bcb2a358472ef4fe30007a1a63"
server_key_iv: "b45f2c66472112af24729e5d0dfd14c7"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.746495 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "25237e76de584d0889f247e65bba9d88"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "218cb4bcb2a358472ef4fe30007a1a63"
server_key_iv: "b45f2c66472112af24729e5d0dfd14c7"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.752254 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 06:57:52.756120 4864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.757692 4850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:52.757876 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "25237e76de584d0889f247e65bba9d88"
format_stamp: "Formatted at 2026-06-20 06:57:52 on dist-test-slave-mz20"
server_key: "218cb4bcb2a358472ef4fe30007a1a63"
server_key_iv: "b45f2c66472112af24729e5d0dfd14c7"
server_key_version: "encryptionkey@0"
I20260620 06:57:52.758082 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:52.776561 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:52.781118 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:52.781384 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:52.782444 4850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:52.784034 4850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:52.784132 4850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.784241 4850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:52.784317 4850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:52.818333 4850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40803
I20260620 06:57:52.818380 4976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40803 every 8 connection(s)
I20260620 06:57:52.819329 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:52.821441 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 4850
I20260620 06:57:52.821635 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:52.822139 388 external_mini_cluster.cc:1428] Setting key 0ba69e969889726d04ded41a2a503049
Jun 20 06:57:52 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:57:52.840570 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.822334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34491 (local address 127.0.97.62:43883)
0620 06:57:52.822600 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.822607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.824266 (+ 1659us) server_negotiation.cc:407] Connection header received
0620 06:57:52.826442 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.826447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.826576 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.826670 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:52.828446 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.829276 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.830956 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.831382 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.831494 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.835141 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:52.835146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:52.835150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:52.835184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:52.837824 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.838459 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.838463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.838466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.838611 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.838998 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.839001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.839003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.839275 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:52.839399 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.840075 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.840283 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 06:57:52.841637 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.823122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43883 (local address 127.0.97.3:34491)
0620 06:57:52.824060 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:52.824123 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:52.826161 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:52.826942 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:52.826952 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:52.827597 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:52.828276 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.828302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.829431 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.829435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.830816 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.830829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.831615 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.831623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.831851 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.832577 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:57:52.832601 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:52.834915 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:52.837938 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:52.837945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:52.837959 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:52.838368 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:52.838699 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:57:52.838702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:57:52.838704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:57:52.838904 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:57:52.839383 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:52.839389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:52.839723 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 06:57:52.839989 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 06:57:52.840402 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":359,"threads_started":1}
I20260620 06:57:52.843027 4977 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43883
I20260620 06:57:52.843327 4977 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:52.844112 4977 heartbeater.cc:511] Master 127.0.97.62:43883 requested a full tablet report, sending...
I20260620 06:57:52.845729 4524 ts_manager.cc:194] Registered new tserver with Master: 25237e76de584d0889f247e65bba9d88 (127.0.97.3:40803)
I20260620 06:57:52.846669 4524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34491
I20260620 06:57:52.850154 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:57:52.868393 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.855907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56002 (local address 127.0.97.62:43883)
0620 06:57:52.856080 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.856086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.856570 (+ 484us) server_negotiation.cc:407] Connection header received
0620 06:57:52.856734 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.856738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.856801 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.856881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:52.858380 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.859178 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.860414 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.860860 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.860977 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.863464 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:52.863469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:52.863472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:52.863503 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:52.865867 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.866491 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.866496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.866499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.866656 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:52.866995 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:52.867001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:52.867004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:52.867318 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:52.867546 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.867873 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.868100 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:57:52.872679 4524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:57:52.879740 4712 heartbeater.cc:503] Master 127.0.97.62:43883 was elected leader, sending a full tablet report...
W20260620 06:57:52.880182 4524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:52.929940 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.893800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56320)
0620 06:57:52.919601 (+ 25801us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:52.919663 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:52.919997 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:52.923252 (+ 3255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:52.923257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:52.923281 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:52.923632 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.923644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.925395 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.925399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.927122 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.927138 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.928814 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.928820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.928994 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929042 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 06:57:52.929209 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 06:57:52.929433 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25413,"mutex_wait_us":24,"thread_start_us":26409,"threads_started":1}
I20260620 06:57:52.930222 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.895254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.97.3:40803)
0620 06:57:52.918420 (+ 23166us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.918434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.921288 (+ 2854us) server_negotiation.cc:407] Connection header received
0620 06:57:52.921596 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.921603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.921891 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.922090 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:52.922981 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.924498 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.927588 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.929010 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.929137 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929346 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.929692 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.929866 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22888,"thread_start_us":23187,"threads_started":1}
I20260620 06:57:52.930567 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.894528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.97.2:41541)
0620 06:57:52.918420 (+ 23892us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.918434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.921145 (+ 2711us) server_negotiation.cc:407] Connection header received
0620 06:57:52.921306 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.921315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.921654 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.921871 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:52.922502 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.924202 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.927343 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.928790 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.928913 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929103 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.930044 (+ 941us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.930216 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23588,"thread_start_us":23899,"threads_started":1}
I20260620 06:57:52.930964 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.894111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.0.97.1:42967)
0620 06:57:52.922342 (+ 28231us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:52.922351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:52.922379 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:57:52.922483 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:52.922490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:52.922755 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:52.922977 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:52.923833 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.925257 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.927300 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.928630 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.928746 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929011 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:52.930427 (+ 1416us) server_negotiation.cc:299] Negotiation successful
0620 06:57:52.930618 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28050,"thread_start_us":27164,"threads_started":1}
I20260620 06:57:52.931028 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.894409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40803 (local address 127.0.0.1:47798)
0620 06:57:52.921152 (+ 26743us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:52.921171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:52.921400 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:52.922420 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:52.922424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:52.922448 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:52.922815 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.922824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.924628 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.924634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.927438 (+ 2804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.927449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.929254 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.929263 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.929451 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929516 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 06:57:52.929613 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 06:57:52.929775 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26621,"mutex_wait_us":136,"thread_start_us":24703,"threads_started":1}
I20260620 06:57:52.931630 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:52.893945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41541 (local address 127.0.0.1:47722)
0620 06:57:52.920462 (+ 26517us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:52.920483 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:52.920665 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:52.921893 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:52.921901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:52.921944 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:52.922313 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.922326 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.924357 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.924363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.927165 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:52.927178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:52.929286 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:52.929292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:52.929524 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:52.929547 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:57:52.929641 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:57:52.929785 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26339,"mutex_wait_us":180,"thread_start_us":26957,"threads_started":1}
I20260620 06:57:52.935315 4912 tablet_service.cc:1467] Processing CreateTablet for tablet c21217a69eb1413698aaee9df055ddae (DEFAULT_TABLE table=test-table [id=99d3753e80ca473cbc7f66a13fb6f069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:52.935330 4647 tablet_service.cc:1467] Processing CreateTablet for tablet c21217a69eb1413698aaee9df055ddae (DEFAULT_TABLE table=test-table [id=99d3753e80ca473cbc7f66a13fb6f069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:52.935330 4780 tablet_service.cc:1467] Processing CreateTablet for tablet c21217a69eb1413698aaee9df055ddae (DEFAULT_TABLE table=test-table [id=99d3753e80ca473cbc7f66a13fb6f069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:52.936829 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21217a69eb1413698aaee9df055ddae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:52.936828 4647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21217a69eb1413698aaee9df055ddae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:52.937250 4912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c21217a69eb1413698aaee9df055ddae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:52.950891 4995 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Bootstrap starting.
I20260620 06:57:52.954916 4995 tablet_bootstrap.cc:654] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:52.956089 4995 log.cc:824] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:52.958009 4996 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Bootstrap starting.
I20260620 06:57:52.961930 4997 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Bootstrap starting.
I20260620 06:57:52.962512 4996 tablet_bootstrap.cc:654] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:52.963827 4996 log.cc:824] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:52.966795 4997 tablet_bootstrap.cc:654] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:52.967028 4996 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: No bootstrap required, opened a new log
I20260620 06:57:52.967305 4996 ts_tablet_manager.cc:1397] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 06:57:52.968276 4997 log.cc:824] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:52.971983 4997 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: No bootstrap required, opened a new log
I20260620 06:57:52.971987 4995 tablet_bootstrap.cc:492] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: No bootstrap required, opened a new log
I20260620 06:57:52.972287 4995 ts_tablet_manager.cc:1397] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20260620 06:57:52.972287 4997 ts_tablet_manager.cc:1397] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 06:57:52.975265 4996 raft_consensus.cc:348] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.975653 4996 raft_consensus.cc:374] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:52.975744 4996 raft_consensus.cc:729] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3869c07ec420a800d19b5db8ce9c9, State: Initialized, Role: FOLLOWER
I20260620 06:57:52.976358 4996 consensus_queue.cc:260] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.977519 4845 heartbeater.cc:503] Master 127.0.97.62:43883 was elected leader, sending a full tablet report...
I20260620 06:57:52.978267 4996 ts_tablet_manager.cc:1428] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 06:57:52.983000 4997 raft_consensus.cc:348] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.982999 4995 raft_consensus.cc:348] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.983459 4995 raft_consensus.cc:374] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:52.983459 4997 raft_consensus.cc:374] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:52.983564 4997 raft_consensus.cc:729] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e1ac3fa03174df69d4ef508d78a38b2, State: Initialized, Role: FOLLOWER
I20260620 06:57:52.983564 4995 raft_consensus.cc:729] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25237e76de584d0889f247e65bba9d88, State: Initialized, Role: FOLLOWER
I20260620 06:57:52.984200 4997 consensus_queue.cc:260] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.984217 4995 consensus_queue.cc:260] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:52.985791 4977 heartbeater.cc:503] Master 127.0.97.62:43883 was elected leader, sending a full tablet report...
I20260620 06:57:52.986097 4997 ts_tablet_manager.cc:1428] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 06:57:52.986267 4995 ts_tablet_manager.cc:1428] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 06:57:53.011804 5001 raft_consensus.cc:482] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:53.012257 5001 raft_consensus.cc:504] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:53.014477 5001 leader_election.cc:288] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e1ac3fa03174df69d4ef508d78a38b2 (127.0.97.1:42967), 25237e76de584d0889f247e65bba9d88 (127.0.97.3:40803)
I20260620 06:57:53.022058 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.014961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55221 (local address 127.0.97.1:42967)
0620 06:57:53.015199 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.015206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.016410 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 06:57:53.016625 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.016630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.016697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.016773 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:53.017366 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.018160 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.020360 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.021141 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.021265 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.021381 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.021651 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.021790 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 06:57:53.022182 5005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.014930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.97.2:55221)
0620 06:57:53.016289 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.016303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:53.016479 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:53.016804 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:53.016809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:53.016831 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:53.017210 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:53.017221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.018286 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.018290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:53.020220 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:53.020233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.021353 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.021359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:53.021501 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.021527 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:57:53.021609 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:57:53.021741 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"mutex_wait_us":373,"spinlock_wait_cycles":2432,"thread_start_us":585,"threads_started":1}
I20260620 06:57:53.022511 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.015145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41783 (local address 127.0.97.3:40803)
0620 06:57:53.015425 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.015432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.016404 (+ 972us) server_negotiation.cc:407] Connection header received
0620 06:57:53.016644 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.016650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.016724 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.016811 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:57:53.017506 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.018328 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.020734 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.021567 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.021686 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.021790 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.022104 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.022243 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 06:57:53.022964 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.014871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40803 (local address 127.0.97.2:41783)
0620 06:57:53.016271 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.016295 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:53.016495 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:53.016831 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:53.016836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:53.016860 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:57:53.017351 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:53.017368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.018484 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.018490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:53.020591 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:53.020603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.021781 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.021787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:53.021951 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.021972 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:57:53.022063 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:57:53.022516 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"spinlock_wait_cycles":51584,"thread_start_us":785,"threads_started":1}
I20260620 06:57:53.023674 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21217a69eb1413698aaee9df055ddae" candidate_uuid: "8df3869c07ec420a800d19b5db8ce9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" is_pre_election: true
I20260620 06:57:53.023919 4932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21217a69eb1413698aaee9df055ddae" candidate_uuid: "8df3869c07ec420a800d19b5db8ce9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25237e76de584d0889f247e65bba9d88" is_pre_election: true
I20260620 06:57:53.024307 4667 raft_consensus.cc:2454] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3869c07ec420a800d19b5db8ce9c9 in term 0.
I20260620 06:57:53.024416 4932 raft_consensus.cc:2454] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3869c07ec420a800d19b5db8ce9c9 in term 0.
I20260620 06:57:53.025269 4733 leader_election.cc:302] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e1ac3fa03174df69d4ef508d78a38b2, 8df3869c07ec420a800d19b5db8ce9c9; no voters:
I20260620 06:57:53.025694 5001 raft_consensus.cc:2781] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:53.025833 5001 raft_consensus.cc:482] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:53.025934 5001 raft_consensus.cc:3037] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:53.028527 5001 raft_consensus.cc:504] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:53.029188 5001 leader_election.cc:288] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [CANDIDATE]: Term 1 election: Requested vote from peers 7e1ac3fa03174df69d4ef508d78a38b2 (127.0.97.1:42967), 25237e76de584d0889f247e65bba9d88 (127.0.97.3:40803)
I20260620 06:57:53.029739 4932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21217a69eb1413698aaee9df055ddae" candidate_uuid: "8df3869c07ec420a800d19b5db8ce9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25237e76de584d0889f247e65bba9d88"
I20260620 06:57:53.029736 4667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c21217a69eb1413698aaee9df055ddae" candidate_uuid: "8df3869c07ec420a800d19b5db8ce9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e1ac3fa03174df69d4ef508d78a38b2"
I20260620 06:57:53.029963 4667 raft_consensus.cc:3037] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:53.030010 4932 raft_consensus.cc:3037] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:53.033315 4667 raft_consensus.cc:2454] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3869c07ec420a800d19b5db8ce9c9 in term 1.
I20260620 06:57:53.033315 4932 raft_consensus.cc:2454] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3869c07ec420a800d19b5db8ce9c9 in term 1.
I20260620 06:57:53.033792 4734 leader_election.cc:302] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 25237e76de584d0889f247e65bba9d88, 7e1ac3fa03174df69d4ef508d78a38b2, 8df3869c07ec420a800d19b5db8ce9c9; no voters:
I20260620 06:57:53.034200 5001 raft_consensus.cc:2781] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:53.034745 5001 raft_consensus.cc:686] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 1 LEADER]: Becoming Leader. State: Replica: 8df3869c07ec420a800d19b5db8ce9c9, State: Running, Role: LEADER
I20260620 06:57:53.035360 5001 consensus_queue.cc:237] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } }
I20260620 06:57:53.040405 4524 catalog_manager.cc:5495] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df3869c07ec420a800d19b5db8ce9c9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "8df3869c07ec420a800d19b5db8ce9c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3869c07ec420a800d19b5db8ce9c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260620 06:57:53.057267 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.051981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.0.97.2:41541)
0620 06:57:53.052154 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.052159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.052177 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:57:53.052281 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.052285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.052377 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.052519 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:57:53.052893 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.053726 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.055141 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.055560 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.055651 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.056083 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:57:53.056718 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:57:53.056833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.056912 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.056983 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:57:53.066664 4932 raft_consensus.cc:1261] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 1 FOLLOWER]: Refusing update from remote peer 8df3869c07ec420a800d19b5db8ce9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:53.066665 4667 raft_consensus.cc:1261] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 1 FOLLOWER]: Refusing update from remote peer 8df3869c07ec420a800d19b5db8ce9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:53.067773 5001 consensus_queue.cc:1035] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e1ac3fa03174df69d4ef508d78a38b2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:53.068140 5006 consensus_queue.cc:1035] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25237e76de584d0889f247e65bba9d88" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:57:53.070837 4978 tablet.cc:2369] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:57:53.072899 4846 tablet.cc:2369] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:57:53.079103 5009 mvcc.cc:205] Tried to move back new op lower bound from 7298820804869550080 to 7298820804756815872. Current Snapshot: MvccSnapshot[applied={T|T < 7298820804869550080}]
I20260620 06:57:53.098304 4523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:53.098642 4523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56002:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:53.102524 4523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31eb6bf4f77e4b29b32c70d878530c54: Sending DeleteTablet for 3 replicas of tablet c21217a69eb1413698aaee9df055ddae
I20260620 06:57:53.103843 4647 tablet_service.cc:1514] Processing DeleteTablet for tablet c21217a69eb1413698aaee9df055ddae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56320
I20260620 06:57:53.103906 4780 tablet_service.cc:1514] Processing DeleteTablet for tablet c21217a69eb1413698aaee9df055ddae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47722
I20260620 06:57:53.103919 4912 tablet_service.cc:1514] Processing DeleteTablet for tablet c21217a69eb1413698aaee9df055ddae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47798
I20260620 06:57:53.105731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4585
I20260620 06:57:53.116523 5018 tablet_replica.cc:330] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: stopping tablet replica
I20260620 06:57:53.117204 5019 tablet_replica.cc:330] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: stopping tablet replica
I20260620 06:57:53.118006 5018 raft_consensus.cc:2229] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:53.118314 5020 tablet_replica.cc:330] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: stopping tablet replica
I20260620 06:57:53.118701 5018 raft_consensus.cc:2258] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:53.118784 5020 raft_consensus.cc:2229] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:53.119500 5019 raft_consensus.cc:2229] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:53.119797 5020 raft_consensus.cc:2258] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:53.119997 5019 raft_consensus.cc:2258] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:53.121873 5018 ts_tablet_manager.cc:1905] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:53.122066 5020 ts_tablet_manager.cc:1905] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:53.129505 5018 ts_tablet_manager.cc:1918] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Jun 20 06:57:53 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:57:53.129689 5018 log.cc:1196] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/c21217a69eb1413698aaee9df055ddae
I20260620 06:57:53.129958 5020 ts_tablet_manager.cc:1918] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:53.130144 5020 log.cc:1196] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/c21217a69eb1413698aaee9df055ddae
I20260620 06:57:53.130249 5018 ts_tablet_manager.cc:1939] T c21217a69eb1413698aaee9df055ddae P 25237e76de584d0889f247e65bba9d88: Deleting consensus metadata
I20260620 06:57:53.130766 5020 ts_tablet_manager.cc:1939] T c21217a69eb1413698aaee9df055ddae P 8df3869c07ec420a800d19b5db8ce9c9: Deleting consensus metadata
I20260620 06:57:53.131873 5019 ts_tablet_manager.cc:1905] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:53.132454 4509 catalog_manager.cc:4841] TS 25237e76de584d0889f247e65bba9d88 (127.0.97.3:40803): tablet c21217a69eb1413698aaee9df055ddae (table test-table [id=99d3753e80ca473cbc7f66a13fb6f069]) successfully deleted
I20260620 06:57:53.132665 4510 catalog_manager.cc:4841] TS 8df3869c07ec420a800d19b5db8ce9c9 (127.0.97.2:41541): tablet c21217a69eb1413698aaee9df055ddae (table test-table [id=99d3753e80ca473cbc7f66a13fb6f069]) successfully deleted
I20260620 06:57:53.139902 5019 ts_tablet_manager.cc:1918] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:53.140107 5019 log.cc:1196] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/c21217a69eb1413698aaee9df055ddae
I20260620 06:57:53.140230 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.107559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.0.97.1:42967)
0620 06:57:53.107847 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.107855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.117332 (+ 9477us) server_negotiation.cc:407] Connection header received
0620 06:57:53.117550 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.117556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.117657 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.117763 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:53.120723 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.121898 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.124431 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.125091 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.125198 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.133788 (+ 8590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:53.133801 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:53.133818 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:53.133867 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:53.136990 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.137768 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.137776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.137778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.137939 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.138333 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.138337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.138339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.138798 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:53.139121 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.139415 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.139671 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 06:57:53.140707 5019 ts_tablet_manager.cc:1939] T c21217a69eb1413698aaee9df055ddae P 7e1ac3fa03174df69d4ef508d78a38b2: Deleting consensus metadata
I20260620 06:57:53.140982 4707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:53.248288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4585
W20260620 06:57:53.270889 4508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42967 (error 108)
I20260620 06:57:53.271170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4718
I20260620 06:57:53.273312 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.272791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56342)
0620 06:57:53.272968 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.273099 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 06:57:53.273622 4508 catalog_manager.cc:4584] TS 7e1ac3fa03174df69d4ef508d78a38b2 (127.0.97.1:42967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c21217a69eb1413698aaee9df055ddae: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Jun 20 06:57:53 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:57:53.293093 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.272762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.0.97.2:41541)
0620 06:57:53.272948 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.272955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.272975 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:57:53.273053 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.273069 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.273161 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.273276 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:53.274955 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.275877 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.277768 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.278702 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.278859 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.284555 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:53.284576 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:53.284603 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:53.284664 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:53.289223 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.290210 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.290222 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.290226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.290436 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.290898 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.290903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.290908 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.291451 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:53.291853 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.292198 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.292465 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:57:53.294291 4840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:53.327625 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.326977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56348)
0620 06:57:53.327253 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.327384 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 06:57:53.328838 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.328369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56356)
0620 06:57:53.328558 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.328677 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:57:53.389967 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.389322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56362)
0620 06:57:53.389624 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.389736 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 06:57:53.391268 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.390896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56368)
0620 06:57:53.391045 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.391132 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:57:53.419842 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4718
I20260620 06:57:53.436884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4850
Jun 20 06:57:53 dist-test-slave-mz20 krb5kdc[4480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:57:53.453656 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.437818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.97.3:40803)
0620 06:57:53.438063 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:53.438070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:53.438094 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:57:53.438259 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:53.438264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:53.438349 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:53.438480 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:53.439869 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.440753 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.442192 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:53.442773 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:53.442886 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:53.447593 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:53.447612 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:53.447633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:57:53.447679 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:53.450809 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.451582 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.451591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.451594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.451786 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:57:53.452219 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:57:53.452222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:57:53.452224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:57:53.452649 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:53.452895 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:53.453145 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:57:53.453346 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:57:53.454370 4972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:53.479187 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.471802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56376)
0620 06:57:53.478793 (+ 6991us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.478945 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6758}
I20260620 06:57:53.480686 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:53.480200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42967 (local address 127.0.0.1:56392)
0620 06:57:53.480434 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:53.480527 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 06:57:53.552356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4850
I20260620 06:57:53.567165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 4493
I20260620 06:57:53.568451 4554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:53.685281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 4493
2026-06-20T06:57:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3994 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T06:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:53Z Disabled control of system clock
I20260620 06:57:53.753043 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34065
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:57:54.105080 5030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:54.105422 5030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:54.105535 5030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:54.112978 5030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:54.113094 5030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:54.113157 5030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:54.113234 5030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:54.113284 5030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:57:54.122627 5030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:54.124256 5030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:54.126449 5030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:54.139690 5035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:54.139726 5038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:54.139690 5036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:54.140656 5030 server_base.cc:1034] running on GCE node
I20260620 06:57:54.141566 5030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:54.143316 5030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:54.144529 5030 hybrid_clock.cc:648] HybridClock initialized: now 1781938674144472 us; error 50 us; skew 500 ppm
I20260620 06:57:54.144897 5030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:54.147467 5030 webserver.cc:460] Webserver started at http://127.0.97.62:34159/ using document root <none> and password file <none>
I20260620 06:57:54.148277 5030 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:54.148379 5030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:54.148748 5030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:54.152189 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "44c54da028724a28b714d4f36b309a9c"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "852de5e30025ee9068d9ca8af6500791"
server_key_iv: "514a15ec824f5cf7d9fa10bd2b3780d0"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.152992 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "44c54da028724a28b714d4f36b309a9c"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "852de5e30025ee9068d9ca8af6500791"
server_key_iv: "514a15ec824f5cf7d9fa10bd2b3780d0"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.158725 5030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 06:57:54.162479 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:54.164033 5030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:54.164206 5030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "44c54da028724a28b714d4f36b309a9c"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "852de5e30025ee9068d9ca8af6500791"
server_key_iv: "514a15ec824f5cf7d9fa10bd2b3780d0"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.164423 5030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:54.170457 5030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:54.171296 5030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:54.171583 5030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:54.189738 5030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34065
I20260620 06:57:54.189759 5095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34065 every 8 connection(s)
I20260620 06:57:54.191295 5030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:54.191871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5030
I20260620 06:57:54.192103 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:54.192938 388 external_mini_cluster.cc:1428] Setting key af07cfc92a0fc4ba42f3e0a0dc7a2dbb
I20260620 06:57:54.196363 5096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:54.202567 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:54.195268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.0.97.62:34065)
0620 06:57:54.196391 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:54.196415 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:54.196480 (+ 65us) server_negotiation.cc:407] Connection header received
0620 06:57:54.198301 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:54.198324 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:54.198842 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:54.199300 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:54.200555 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:54.200563 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:54.200578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:54.200597 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:54.200701 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:54.200726 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:54.200972 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:54.201143 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 06:57:54.201442 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":384,"threads_started":1}
I20260620 06:57:54.208869 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: Bootstrap starting.
I20260620 06:57:54.212802 5096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:54.213980 5096 log.cc:824] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:54.217439 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: No bootstrap required, opened a new log
I20260620 06:57:54.225397 5096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } }
I20260620 06:57:54.225869 5096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:54.225980 5096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c54da028724a28b714d4f36b309a9c, State: Initialized, Role: FOLLOWER
I20260620 06:57:54.226739 5096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } }
I20260620 06:57:54.227025 5096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:54.227147 5096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:54.227290 5096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:54.229476 5096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } }
I20260620 06:57:54.229991 5096 leader_election.cc:302] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44c54da028724a28b714d4f36b309a9c; no voters:
I20260620 06:57:54.230652 5096 leader_election.cc:288] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:54.230801 5101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:54.231526 5101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [term 1 LEADER]: Becoming Leader. State: Replica: 44c54da028724a28b714d4f36b309a9c, State: Running, Role: LEADER
I20260620 06:57:54.232146 5101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } }
I20260620 06:57:54.232908 5096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:54.235126 5102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44c54da028724a28b714d4f36b309a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } } }
I20260620 06:57:54.235514 5102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:54.236035 5103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44c54da028724a28b714d4f36b309a9c. Latest consensus state: current_term: 1 leader_uuid: "44c54da028724a28b714d4f36b309a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44c54da028724a28b714d4f36b309a9c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34065 } } }
I20260620 06:57:54.236362 5103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:54.236457 5109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:54.245141 5109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:54.256078 5109 catalog_manager.cc:1347] Generated new cluster ID: 5e3c1f9c2b3d4f33877714d5c2dc2504
I20260620 06:57:54.256222 5109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:54.277802 5109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:54.278316 5109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:54.290417 5109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: Generated new TSK 0
I20260620 06:57:54.291859 5109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:57:54.306993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:57:54.680369 5120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:54.680708 5120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:54.680833 5120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:54.688614 5120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:54.688732 5120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:54.688889 5120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:57:54.698755 5120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:54.700393 5120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:54.702520 5120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:54.716764 5125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:54.716830 5128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:54.716807 5126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:54.717607 5120 server_base.cc:1034] running on GCE node
I20260620 06:57:54.718312 5120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:54.719491 5120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:54.720728 5120 hybrid_clock.cc:648] HybridClock initialized: now 1781938674720657 us; error 51 us; skew 500 ppm
I20260620 06:57:54.721123 5120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:54.723923 5120 webserver.cc:460] Webserver started at http://127.0.97.1:38809/ using document root <none> and password file <none>
I20260620 06:57:54.724752 5120 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:54.724851 5120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:54.725209 5120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:54.728672 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "33a1bea5031c4039909a174f1b170779"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "a9ce8d18f1be0e5326557f4ca3d0c309"
server_key_iv: "7dd690201e0dc1e7126a4806e1a389a9"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.729588 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33a1bea5031c4039909a174f1b170779"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "a9ce8d18f1be0e5326557f4ca3d0c309"
server_key_iv: "7dd690201e0dc1e7126a4806e1a389a9"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.735991 5120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 06:57:54.740304 5134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:54.741972 5120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260620 06:57:54.742177 5120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "33a1bea5031c4039909a174f1b170779"
format_stamp: "Formatted at 2026-06-20 06:57:54 on dist-test-slave-mz20"
server_key: "a9ce8d18f1be0e5326557f4ca3d0c309"
server_key_iv: "7dd690201e0dc1e7126a4806e1a389a9"
server_key_version: "encryptionkey@0"
I20260620 06:57:54.742425 5120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:54.748725 5120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:54.749632 5120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:54.749902 5120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:54.750942 5120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:54.752501 5120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:54.752602 5120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:54.752712 5120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:54.752784 5120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:54.786470 5120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40881
I20260620 06:57:54.786514 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40881 every 8 connection(s)
I20260620 06:57:54.788044 5120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:57:54.790205 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5120
I20260620 06:57:54.790419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:57:54.791316 388 external_mini_cluster.cc:1428] Setting key 83e4a732db9424790c7f556689fae923
I20260620 06:57:54.794569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 06:57:54.798383 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:54.791312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38287 (local address 127.0.97.62:34065)
0620 06:57:54.791784 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:54.791789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:54.793611 (+ 1822us) server_negotiation.cc:407] Connection header received
0620 06:57:54.795621 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:54.795627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:54.795704 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:54.795828 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:54.797431 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:54.797438 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:54.797444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:54.797465 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:54.797522 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:54.797539 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:54.797668 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:54.797973 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 06:57:54.798051 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
I20260620 06:57:54.799861 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:54.792190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34065 (local address 127.0.97.1:38287)
0620 06:57:54.793447 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:54.793506 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:54.795307 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:54.796085 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:54.796095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:54.796228 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:54.797001 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:54.797004 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:54.797065 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:54.797214 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:54.797720 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:54.797732 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:54.797740 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:57:54.798050 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 06:57:54.798382 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":373,"threads_started":1}
I20260620 06:57:54.801801 5247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34065
I20260620 06:57:54.802174 5247 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:54.802939 5247 heartbeater.cc:511] Master 127.0.97.62:34065 requested a full tablet report, sending...
I20260620 06:57:54.805365 5061 ts_manager.cc:194] Registered new tserver with Master: 33a1bea5031c4039909a174f1b170779 (127.0.97.1:40881)
W20260620 06:57:55.167855 5251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:55.168201 5251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:55.168344 5251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:55.176282 5251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:55.176412 5251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:55.176579 5251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:57:55.186486 5251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:55.188237 5251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:55.191025 5251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:55.207180 5256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:55.207269 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:55.207518 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:55.208351 5251 server_base.cc:1034] running on GCE node
I20260620 06:57:55.208993 5251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:55.210038 5251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:55.211237 5251 hybrid_clock.cc:648] HybridClock initialized: now 1781938675211180 us; error 45 us; skew 500 ppm
I20260620 06:57:55.211604 5251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:55.214156 5251 webserver.cc:460] Webserver started at http://127.0.97.2:43387/ using document root <none> and password file <none>
I20260620 06:57:55.214984 5251 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:55.215075 5251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:55.215466 5251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:55.218747 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "b74cbea29fa3adfcda975cc3f8a523d3"
server_key_iv: "984e4c5d6929d485d8cf575514985d2b"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.219587 5251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "b74cbea29fa3adfcda975cc3f8a523d3"
server_key_iv: "984e4c5d6929d485d8cf575514985d2b"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.225262 5251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:57:55.229305 5265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.230950 5251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:57:55.231110 5251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "b74cbea29fa3adfcda975cc3f8a523d3"
server_key_iv: "984e4c5d6929d485d8cf575514985d2b"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.231346 5251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:55.237881 5251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:55.238996 5251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:55.239321 5251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:55.240396 5251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:55.241981 5251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:55.242076 5251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.242197 5251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:55.242273 5251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.274457 5251 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42519
I20260620 06:57:55.274499 5377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42519 every 8 connection(s)
I20260620 06:57:55.276044 5251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:57:55.278954 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5251
I20260620 06:57:55.279170 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:57:55.279718 388 external_mini_cluster.cc:1428] Setting key 9d669488b58987d6f0bd76e9d28f09f9
I20260620 06:57:55.283396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--builtin_ntp_servers=127.0.97.20:46217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 06:57:55.286237 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.279269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52919 (local address 127.0.97.62:34065)
0620 06:57:55.279506 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.279514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.281441 (+ 1927us) server_negotiation.cc:407] Connection header received
0620 06:57:55.283584 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.283590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.283669 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.283759 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.285455 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.285460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.285463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.285482 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.285518 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.285537 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.285662 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.285859 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.285940 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:57:55.287468 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.279918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34065 (local address 127.0.97.2:52919)
0620 06:57:55.281231 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.281293 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.283260 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.284070 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.284080 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.284201 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.285062 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.285064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.285113 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.285242 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.285635 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.285647 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.285655 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.285960 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.286288 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":378,"threads_started":1}
I20260620 06:57:55.289155 5378 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34065
I20260620 06:57:55.289510 5378 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:55.290373 5378 heartbeater.cc:511] Master 127.0.97.62:34065 requested a full tablet report, sending...
I20260620 06:57:55.292332 5061 ts_manager.cc:194] Registered new tserver with Master: 30eaf0dae36e40e7a1761cb1c9e09ff6 (127.0.97.2:42519)
W20260620 06:57:55.654321 5381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:55.654675 5381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:55.654805 5381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:55.661904 5381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:55.662020 5381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:55.662173 5381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:57:55.671257 5381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:55.672765 5381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:55.674808 5381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:55.688234 5387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:55.688287 5386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:55.688686 5381 server_base.cc:1034] running on GCE node
W20260620 06:57:55.688333 5389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:55.689455 5381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:55.690346 5381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:55.691586 5381 hybrid_clock.cc:648] HybridClock initialized: now 1781938675691496 us; error 59 us; skew 500 ppm
I20260620 06:57:55.691936 5381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:55.694221 5381 webserver.cc:460] Webserver started at http://127.0.97.3:37779/ using document root <none> and password file <none>
I20260620 06:57:55.695029 5381 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:55.695130 5381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:55.695436 5381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:55.698381 5381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "5f0795fcd1ea47b6a96edc96123a3ba6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "81b4809ca8423b54bf57977b4f4bc95a"
server_key_iv: "7006e276af1d9618cff1321ceb1641d8"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.699193 5381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5f0795fcd1ea47b6a96edc96123a3ba6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "81b4809ca8423b54bf57977b4f4bc95a"
server_key_iv: "7006e276af1d9618cff1321ceb1641d8"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.704316 5381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:57:55.707821 5395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.709359 5381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:55.709529 5381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "5f0795fcd1ea47b6a96edc96123a3ba6"
format_stamp: "Formatted at 2026-06-20 06:57:55 on dist-test-slave-mz20"
server_key: "81b4809ca8423b54bf57977b4f4bc95a"
server_key_iv: "7006e276af1d9618cff1321ceb1641d8"
server_key_version: "encryptionkey@0"
I20260620 06:57:55.709745 5381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:55.715451 5381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:55.716238 5381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:55.716481 5381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:55.717403 5381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:57:55.718801 5381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:57:55.718894 5381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.719007 5381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:57:55.719075 5381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:55.749497 5381 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37683
I20260620 06:57:55.749554 5507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37683 every 8 connection(s)
I20260620 06:57:55.750964 5381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:57:55.757835 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5381
I20260620 06:57:55.758060 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:57:55.758625 388 external_mini_cluster.cc:1428] Setting key ab9eaab68268117e957dbd516561e370
I20260620 06:57:55.759029 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.753657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48933 (local address 127.0.97.62:34065)
0620 06:57:55.753841 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.753846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.755271 (+ 1425us) server_negotiation.cc:407] Connection header received
0620 06:57:55.756857 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.756862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.756925 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.757016 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.758319 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.758324 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.758327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.758343 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.758376 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.758390 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.758486 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.758726 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.758791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:57:55.760116 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.754173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34065 (local address 127.0.97.3:48933)
0620 06:57:55.755122 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.755158 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.756610 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.757212 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.757221 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.757314 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.757985 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.757988 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.758038 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.758152 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.758483 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.758491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.758498 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.758804 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.759069 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":279,"threads_started":1}
I20260620 06:57:55.761286 5508 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34065
I20260620 06:57:55.761571 5508 heartbeater.cc:464] Registering TS with master...
I20260620 06:57:55.762159 5508 heartbeater.cc:511] Master 127.0.97.62:34065 requested a full tablet report, sending...
I20260620 06:57:55.763586 5060 ts_manager.cc:194] Registered new tserver with Master: 5f0795fcd1ea47b6a96edc96123a3ba6 (127.0.97.3:37683)
I20260620 06:57:55.774015 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:57:55.781358 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.778670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.0.97.62:34065)
0620 06:57:55.778864 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.778872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.779404 (+ 532us) server_negotiation.cc:407] Connection header received
0620 06:57:55.779512 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.779516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.779571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.779667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.780725 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.780729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.780732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.780748 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.780791 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.780804 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.780912 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.781011 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.781077 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:57:55.785800 5060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51030:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:57:55.792888 5060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:57:55.808936 5247 heartbeater.cc:503] Master 127.0.97.62:34065 was elected leader, sending a full tablet report...
I20260620 06:57:55.826417 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.806151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.0.97.1:40881)
0620 06:57:55.822980 (+ 16829us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.822988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.823021 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:57:55.823099 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.823108 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.823395 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.824211 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.825542 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.825550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.825563 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.825579 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.825625 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.825640 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.825839 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.825908 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.826072 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16665,"thread_start_us":16670,"threads_started":1}
I20260620 06:57:55.826462 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.806273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.0.97.3:37683)
0620 06:57:55.821815 (+ 15542us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.821835 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.823082 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 06:57:55.823356 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.823366 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.823594 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.823737 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.825501 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.825519 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.825534 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.825550 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.825617 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.825638 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.825793 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.826000 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.826101 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15328,"thread_start_us":15601,"threads_started":1}
I20260620 06:57:55.826449 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.805540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46622)
0620 06:57:55.806302 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.806338 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.806480 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.823625 (+ 17145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.823633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.823734 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.824547 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.824550 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.824578 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.824658 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.825743 (+ 1085us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.825756 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.825764 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.825927 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.826014 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":780,"threads_started":1}
I20260620 06:57:55.826884 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.805984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37683 (local address 127.0.0.1:37364)
0620 06:57:55.822954 (+ 16970us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.822974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.823190 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.824756 (+ 1566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.824759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.824776 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.825311 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.825313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.825327 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.825363 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.825845 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.825849 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.825852 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.826375 (+ 523us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.826429 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16827,"thread_start_us":17432,"threads_started":1}
I20260620 06:57:55.827656 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.807465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.0.97.2:42519)
0620 06:57:55.822439 (+ 14974us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.822452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.823775 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 06:57:55.824012 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.824021 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.824227 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.824340 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.826800 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.826808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.826821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.826838 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.826896 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.826920 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.827057 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.827245 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.827344 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14835,"thread_start_us":14723,"threads_started":1}
I20260620 06:57:55.828508 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.807162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42519 (local address 127.0.0.1:58188)
0620 06:57:55.823675 (+ 16513us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.823690 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.823863 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.825216 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.825221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.825241 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.826601 (+ 1360us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.826605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.826620 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.826665 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.827106 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.827111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.827116 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.828128 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.828178 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16395,"thread_start_us":15459,"threads_started":1}
I20260620 06:57:55.830870 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b (DEFAULT_TABLE table=test-table [id=ad61b3bfd86d4d04aed7327fd94dff8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:55.831224 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b (DEFAULT_TABLE table=test-table [id=ad61b3bfd86d4d04aed7327fd94dff8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:55.832154 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809e39c6bdff405d82c6fbcad5a5662b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:55.831944 5313 tablet_service.cc:1467] Processing CreateTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b (DEFAULT_TABLE table=test-table [id=ad61b3bfd86d4d04aed7327fd94dff8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:57:55.832608 5443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809e39c6bdff405d82c6fbcad5a5662b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:55.833187 5313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 809e39c6bdff405d82c6fbcad5a5662b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:55.843544 5526 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Bootstrap starting.
I20260620 06:57:55.846909 5527 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Bootstrap starting.
I20260620 06:57:55.846982 5526 tablet_bootstrap.cc:654] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:55.848266 5526 log.cc:824] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:55.848568 5528 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Bootstrap starting.
I20260620 06:57:55.850775 5527 tablet_bootstrap.cc:654] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:55.851605 5528 tablet_bootstrap.cc:654] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:55.852030 5527 log.cc:824] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:55.852545 5528 log.cc:824] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:55.855052 5528 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: No bootstrap required, opened a new log
I20260620 06:57:55.855110 5527 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: No bootstrap required, opened a new log
I20260620 06:57:55.855114 5526 tablet_bootstrap.cc:492] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: No bootstrap required, opened a new log
I20260620 06:57:55.855329 5528 ts_tablet_manager.cc:1397] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 06:57:55.855342 5527 ts_tablet_manager.cc:1397] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 06:57:55.856408 5526 ts_tablet_manager.cc:1397] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 06:57:55.862164 5528 raft_consensus.cc:348] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.862520 5528 raft_consensus.cc:374] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:55.862684 5528 raft_consensus.cc:729] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0795fcd1ea47b6a96edc96123a3ba6, State: Initialized, Role: FOLLOWER
I20260620 06:57:55.863308 5528 consensus_queue.cc:260] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.864332 5508 heartbeater.cc:503] Master 127.0.97.62:34065 was elected leader, sending a full tablet report...
I20260620 06:57:55.865000 5528 ts_tablet_manager.cc:1428] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260620 06:57:55.865185 5527 raft_consensus.cc:348] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.865572 5527 raft_consensus.cc:374] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:55.865718 5527 raft_consensus.cc:729] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30eaf0dae36e40e7a1761cb1c9e09ff6, State: Initialized, Role: FOLLOWER
I20260620 06:57:55.866281 5527 consensus_queue.cc:260] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.867347 5378 heartbeater.cc:503] Master 127.0.97.62:34065 was elected leader, sending a full tablet report...
I20260620 06:57:55.868019 5527 ts_tablet_manager.cc:1428] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 06:57:55.868137 5526 raft_consensus.cc:348] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.868536 5526 raft_consensus.cc:374] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:55.868623 5526 raft_consensus.cc:729] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a1bea5031c4039909a174f1b170779, State: Initialized, Role: FOLLOWER
I20260620 06:57:55.869320 5526 consensus_queue.cc:260] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.871693 5526 ts_tablet_manager.cc:1428] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260620 06:57:55.890686 5533 raft_consensus.cc:482] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:57:55.890971 5533 raft_consensus.cc:504] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.892831 5533 leader_election.cc:288] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33a1bea5031c4039909a174f1b170779 (127.0.97.1:40881), 5f0795fcd1ea47b6a96edc96123a3ba6 (127.0.97.3:37683)
I20260620 06:57:55.896114 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.893206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51595 (local address 127.0.97.3:37683)
0620 06:57:55.893393 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.893399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.894246 (+ 847us) server_negotiation.cc:407] Connection header received
0620 06:57:55.894397 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.894400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.894446 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.894522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.895530 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.895536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.895540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.895556 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.895584 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.895596 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.895699 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.895811 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.895869 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 06:57:55.896198 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.893201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59747 (local address 127.0.97.1:40881)
0620 06:57:55.893382 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.893388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.894202 (+ 814us) server_negotiation.cc:407] Connection header received
0620 06:57:55.894343 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.894348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.894412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.894522 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.895634 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.895639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.895642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.895657 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.895685 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.895696 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.895801 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.895903 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.895959 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:57:55.896349 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.893447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37683 (local address 127.0.97.2:51595)
0620 06:57:55.894141 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.894156 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.894268 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.894582 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.894586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.894607 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.895341 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.895344 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.895358 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.895397 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.895698 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.895702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.895707 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.895861 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.895908 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":383,"threads_started":1}
I20260620 06:57:55.896541 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.893270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.97.2:59747)
0620 06:57:55.894083 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:55.894100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:55.894220 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:55.894559 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:55.894563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:55.894586 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:57:55.895435 (+ 849us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:57:55.895438 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:57:55.895450 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:57:55.895484 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:57:55.895781 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:57:55.895786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:57:55.895791 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:57:55.895926 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 06:57:55.895972 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":459,"threads_started":1}
I20260620 06:57:55.897248 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809e39c6bdff405d82c6fbcad5a5662b" candidate_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" is_pre_election: true
I20260620 06:57:55.897326 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809e39c6bdff405d82c6fbcad5a5662b" candidate_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a1bea5031c4039909a174f1b170779" is_pre_election: true
I20260620 06:57:55.897709 5463 raft_consensus.cc:2454] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30eaf0dae36e40e7a1761cb1c9e09ff6 in term 0.
I20260620 06:57:55.897804 5202 raft_consensus.cc:2454] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30eaf0dae36e40e7a1761cb1c9e09ff6 in term 0.
I20260620 06:57:55.898288 5266 leader_election.cc:302] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30eaf0dae36e40e7a1761cb1c9e09ff6, 5f0795fcd1ea47b6a96edc96123a3ba6; no voters:
I20260620 06:57:55.898635 5533 raft_consensus.cc:2781] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:57:55.898773 5533 raft_consensus.cc:482] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:57:55.898866 5533 raft_consensus.cc:3037] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:55.900929 5533 raft_consensus.cc:504] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.901463 5533 leader_election.cc:288] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 33a1bea5031c4039909a174f1b170779 (127.0.97.1:40881), 5f0795fcd1ea47b6a96edc96123a3ba6 (127.0.97.3:37683)
I20260620 06:57:55.901896 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809e39c6bdff405d82c6fbcad5a5662b" candidate_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a1bea5031c4039909a174f1b170779"
I20260620 06:57:55.901942 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "809e39c6bdff405d82c6fbcad5a5662b" candidate_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6"
I20260620 06:57:55.902076 5202 raft_consensus.cc:3037] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:55.902091 5463 raft_consensus.cc:3037] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:55.904193 5463 raft_consensus.cc:2454] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30eaf0dae36e40e7a1761cb1c9e09ff6 in term 1.
I20260620 06:57:55.904229 5202 raft_consensus.cc:2454] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30eaf0dae36e40e7a1761cb1c9e09ff6 in term 1.
I20260620 06:57:55.904525 5266 leader_election.cc:302] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30eaf0dae36e40e7a1761cb1c9e09ff6, 5f0795fcd1ea47b6a96edc96123a3ba6; no voters:
I20260620 06:57:55.904786 5533 raft_consensus.cc:2781] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:55.905195 5533 raft_consensus.cc:686] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 30eaf0dae36e40e7a1761cb1c9e09ff6, State: Running, Role: LEADER
I20260620 06:57:55.905740 5533 consensus_queue.cc:237] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } }
I20260620 06:57:55.910399 5061 catalog_manager.cc:5495] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30eaf0dae36e40e7a1761cb1c9e09ff6 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30eaf0dae36e40e7a1761cb1c9e09ff6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42519 } health_report { overall_health: HEALTHY } } }
I20260620 06:57:55.937698 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.935647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.97.2:42519)
0620 06:57:55.935845 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.935851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.935893 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:57:55.936116 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.936120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.936196 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.936278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.937112 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.937116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.937119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.937134 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.937164 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.937176 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.937269 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.937361 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.937405 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 06:57:55.945842 5463 raft_consensus.cc:1261] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 1 FOLLOWER]: Refusing update from remote peer 30eaf0dae36e40e7a1761cb1c9e09ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:55.945847 5202 raft_consensus.cc:1261] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 1 FOLLOWER]: Refusing update from remote peer 30eaf0dae36e40e7a1761cb1c9e09ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:57:55.946923 5533 consensus_queue.cc:1035] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a1bea5031c4039909a174f1b170779" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:55.947249 5537 consensus_queue.cc:1035] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0795fcd1ea47b6a96edc96123a3ba6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:57:55.974103 5059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:55.974423 5059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:57:55.978044 5059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44c54da028724a28b714d4f36b309a9c: Sending DeleteTablet for 3 replicas of tablet 809e39c6bdff405d82c6fbcad5a5662b
I20260620 06:57:55.979147 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:55 UTC) from {username='slave'} at 127.0.0.1:37364
I20260620 06:57:55.979615 5182 tablet_service.cc:1514] Processing DeleteTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:55 UTC) from {username='slave'} at 127.0.0.1:46622
I20260620 06:57:55.979759 5313 tablet_service.cc:1514] Processing DeleteTablet for tablet 809e39c6bdff405d82c6fbcad5a5662b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:57:55 UTC) from {username='slave'} at 127.0.0.1:58188
I20260620 06:57:55.980443 5551 tablet_replica.cc:330] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: stopping tablet replica
I20260620 06:57:55.980804 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5120
I20260620 06:57:55.980877 5551 raft_consensus.cc:2229] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:55.981323 5551 raft_consensus.cc:2258] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:55.984023 5551 ts_tablet_manager.cc:1905] T 809e39c6bdff405d82c6fbcad5a5662b P 33a1bea5031c4039909a174f1b170779: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:55.986028 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:55.981927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.0.97.1:40881)
0620 06:57:55.982097 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:55.982104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:55.982294 (+ 190us) server_negotiation.cc:407] Connection header received
0620 06:57:55.982355 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:55.982360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:55.982441 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:55.982577 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:55.985230 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:55.985236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:55.985240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:55.985259 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:55.985298 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:55.985314 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:55.985413 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:55.985657 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 06:57:55.985720 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:57:55.987119 5242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:55.987418 5552 tablet_replica.cc:330] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: stopping tablet replica
I20260620 06:57:55.987876 5552 raft_consensus.cc:2229] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:57:55.988583 5552 raft_consensus.cc:2258] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:55.990513 5552 ts_tablet_manager.cc:1905] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:55.995007 5550 tablet_replica.cc:330] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: stopping tablet replica
I20260620 06:57:55.995416 5550 raft_consensus.cc:2229] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:57:55.995882 5550 raft_consensus.cc:2258] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:57:55.997560 5552 ts_tablet_manager.cc:1918] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:55.997779 5552 log.cc:1196] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/809e39c6bdff405d82c6fbcad5a5662b
I20260620 06:57:55.997808 5550 ts_tablet_manager.cc:1905] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:57:55.998395 5552 ts_tablet_manager.cc:1939] T 809e39c6bdff405d82c6fbcad5a5662b P 30eaf0dae36e40e7a1761cb1c9e09ff6: Deleting consensus metadata
I20260620 06:57:56.000275 5046 catalog_manager.cc:4841] TS 30eaf0dae36e40e7a1761cb1c9e09ff6 (127.0.97.2:42519): tablet 809e39c6bdff405d82c6fbcad5a5662b (table test-table [id=ad61b3bfd86d4d04aed7327fd94dff8d]) successfully deleted
I20260620 06:57:56.004508 5550 ts_tablet_manager.cc:1918] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:57:56.004659 5550 log.cc:1196] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/809e39c6bdff405d82c6fbcad5a5662b
I20260620 06:57:56.005079 5550 ts_tablet_manager.cc:1939] T 809e39c6bdff405d82c6fbcad5a5662b P 5f0795fcd1ea47b6a96edc96123a3ba6: Deleting consensus metadata
I20260620 06:57:56.006525 5045 catalog_manager.cc:4841] TS 5f0795fcd1ea47b6a96edc96123a3ba6 (127.0.97.3:37683): tablet 809e39c6bdff405d82c6fbcad5a5662b (table test-table [id=ad61b3bfd86d4d04aed7327fd94dff8d]) successfully deleted
I20260620 06:57:56.094559 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5120
W20260620 06:57:56.108158 5046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40881 (error 108)
I20260620 06:57:56.108347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5251
I20260620 06:57:56.110165 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.109696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46650)
0620 06:57:56.109868 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.109993 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 06:57:56.110410 5046 catalog_manager.cc:4584] TS 33a1bea5031c4039909a174f1b170779 (127.0.97.1:40881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 809e39c6bdff405d82c6fbcad5a5662b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
I20260620 06:57:56.111716 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.109395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.0.97.2:42519)
0620 06:57:56.109594 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:56.109600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:56.109619 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:57:56.109700 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:56.109705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:56.109763 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:56.109859 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:56.110985 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:56.110991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:56.110996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:56.111016 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:56.111057 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:56.111073 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:56.111200 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:56.111348 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 06:57:56.111408 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:57:56.112433 5373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:56.163892 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.163423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46658)
0620 06:57:56.163638 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.163725 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 06:57:56.164851 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.164527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46662)
0620 06:57:56.164674 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.164742 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:57:56.219297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5251
I20260620 06:57:56.225395 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.224903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46666)
0620 06:57:56.225106 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.225223 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 06:57:56.226351 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.226016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46680)
0620 06:57:56.226158 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.226229 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:57:56.234082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5381
I20260620 06:57:56.236843 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.234771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.97.3:37683)
0620 06:57:56.234958 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:56.234963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:56.235022 (+ 59us) server_negotiation.cc:407] Connection header received
0620 06:57:56.235182 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:56.235186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:56.235283 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:56.235366 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:56.236335 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:56.236338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:56.236341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:56.236353 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:56.236377 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:56.236388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:56.236484 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:56.236522 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 06:57:56.236566 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:57:56.237504 5503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:56.307425 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.306930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46684)
0620 06:57:56.307167 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.307260 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 06:57:56.308398 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.308024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40881 (local address 127.0.0.1:46686)
0620 06:57:56.308222 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:56.308286 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 06:57:56.324429 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5381
I20260620 06:57:56.337919 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5030
I20260620 06:57:56.339088 5091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:56.443518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5030
2026-06-20T06:57:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2754 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 06:57:56.480592 388 mini_oidc.cc:152] Starting JWKS server
I20260620 06:57:56.483129 388 webserver.cc:460] Webserver started at https://127.0.0.1:33797/ using document root <none> and password file <none>
I20260620 06:57:56.483421 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:57:56.484093 388 webserver.cc:460] Webserver started at http://127.0.0.1:38499/ using document root <none> and password file <none>
2026-06-20T06:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:56Z Disabled control of system clock
I20260620 06:57:56.507406 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44473
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44473
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33797/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/testchainca.cert with env {}
W20260620 06:57:56.849325 5564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:56.849629 5564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:56.849741 5564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:56.857105 5564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:56.857244 5564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:56.857306 5564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:56.857381 5564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:56.857445 5564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:57:56.857527 5564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:57:56.857573 5564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33797/jwks.json
I20260620 06:57:56.867525 5564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33797/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:56.868777 5564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:56.870008 5564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:56.880952 5569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:56.880983 5570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:56.881027 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:56.881606 5564 server_base.cc:1034] running on GCE node
I20260620 06:57:56.882375 5564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:56.883693 5564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:56.884923 5564 hybrid_clock.cc:648] HybridClock initialized: now 1781938676884867 us; error 49 us; skew 500 ppm
I20260620 06:57:56.885327 5564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:56.887569 5564 webserver.cc:460] Webserver started at http://127.0.97.62:38717/ using document root <none> and password file <none>
I20260620 06:57:56.888221 5564 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:56.888309 5564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:56.888585 5564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:56.891446 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "c8765181f96b4a40bf0791d73f83a7b0"
format_stamp: "Formatted at 2026-06-20 06:57:56 on dist-test-slave-mz20"
server_key: "05cf989865506e6c7665b29b9ac7407e"
server_key_iv: "9d84a2ef46b3c1c8f240f67879899f03"
server_key_version: "encryptionkey@0"
I20260620 06:57:56.892277 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "c8765181f96b4a40bf0791d73f83a7b0"
format_stamp: "Formatted at 2026-06-20 06:57:56 on dist-test-slave-mz20"
server_key: "05cf989865506e6c7665b29b9ac7407e"
server_key_iv: "9d84a2ef46b3c1c8f240f67879899f03"
server_key_version: "encryptionkey@0"
I20260620 06:57:56.897748 5564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:57:56.901444 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:56.903044 5564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:57:56.903229 5564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "c8765181f96b4a40bf0791d73f83a7b0"
format_stamp: "Formatted at 2026-06-20 06:57:56 on dist-test-slave-mz20"
server_key: "05cf989865506e6c7665b29b9ac7407e"
server_key_iv: "9d84a2ef46b3c1c8f240f67879899f03"
server_key_version: "encryptionkey@0"
I20260620 06:57:56.903434 5564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:56.927975 5564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:56.928812 5564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:56.929077 5564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:56.946074 5564 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44473
I20260620 06:57:56.946095 5629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44473 every 8 connection(s)
I20260620 06:57:56.947391 5564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:56.951064 5630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:56.956539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5564
I20260620 06:57:56.956727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:56.957484 388 external_mini_cluster.cc:1428] Setting key 2fe5b2b24f7a44465c4f98b1b0ed6a54
I20260620 06:57:56.962340 5630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0: Bootstrap starting.
I20260620 06:57:56.965281 5630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:56.966200 5630 log.cc:824] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:56.968765 5630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0: No bootstrap required, opened a new log
I20260620 06:57:56.970271 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:56.959531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.0.97.62:44473)
0620 06:57:56.960545 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:56.960556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:56.960596 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:57:56.962182 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:56.962204 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:56.962727 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:56.963069 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:56.963590 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:56.964862 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:56.966176 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:56.966921 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:56.967099 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:56.968190 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:56.968197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:56.968211 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:56.968707 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:56.968781 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:56.968800 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:56.968893 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:56.969123 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 06:57:56.969405 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260620 06:57:56.973857 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:57:56.976389 5630 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } }
I20260620 06:57:56.976729 5630 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:56.976831 5630 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8765181f96b4a40bf0791d73f83a7b0, State: Initialized, Role: FOLLOWER
I20260620 06:57:56.977365 5630 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } }
I20260620 06:57:56.977563 5630 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:56.977689 5630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:56.977838 5630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:56.979908 5630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } }
I20260620 06:57:56.980346 5630 leader_election.cc:302] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8765181f96b4a40bf0791d73f83a7b0; no voters:
I20260620 06:57:56.980887 5630 leader_election.cc:288] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:56.981062 5635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:56.981741 5635 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [term 1 LEADER]: Becoming Leader. State: Replica: c8765181f96b4a40bf0791d73f83a7b0, State: Running, Role: LEADER
I20260620 06:57:56.982231 5635 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } }
I20260620 06:57:56.982935 5630 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:56.984908 5636 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8765181f96b4a40bf0791d73f83a7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } } }
I20260620 06:57:56.985417 5636 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:56.985016 5637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8765181f96b4a40bf0791d73f83a7b0. Latest consensus state: current_term: 1 leader_uuid: "c8765181f96b4a40bf0791d73f83a7b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8765181f96b4a40bf0791d73f83a7b0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44473 } } }
I20260620 06:57:56.985636 5637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:56.986980 5644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:56.991549 5644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:57.000594 5644 catalog_manager.cc:1347] Generated new cluster ID: 2c505ed3841c4e34834bffef8ed1022f
I20260620 06:57:57.000698 5644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:57.014918 5644 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:57.016156 5644 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:57.021489 5644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8765181f96b4a40bf0791d73f83a7b0: Generated new TSK 0
I20260620 06:57:57.022349 5644 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:57:57.055088 5660 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 06:57:57.055239 5660 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:57:57.056429 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:57.034901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.0.97.62:44473)
0620 06:57:57.035093 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:57.035100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:57.035550 (+ 450us) server_negotiation.cc:407] Connection header received
0620 06:57:57.035854 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:57.035859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:57.035949 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:57.036097 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:57:57.036592 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.037704 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.039200 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.039844 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.039994 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:57.040483 (+ 489us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:57:57.055769 (+ 15286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:57:57.056138 (+ 369us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 06:57:57.056543 5633 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59986: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 06:57:57.058151 5659 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:57:57.034755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44473 (local address 127.0.0.1:59986)
0620 06:57:57.035428 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:57.035443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:57.035614 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:57.036128 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:57.036131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:57.036140 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:57:57.036446 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:57.036454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.037858 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.037863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:57.039044 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:57.039056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.039977 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.040029 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:57.040150 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:57.040207 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:57:57.056211 (+ 16004us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:57:57.056673 (+ 462us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":363,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:44473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 06:57:57.059931 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5564
I20260620 06:57:57.061033 5625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:57.151420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5564
2026-06-20T06:57:57Z chronyd exiting
I20260620 06:57:57.198207 388 test_util.cc:175] -----------------------------------------------
I20260620 06:57:57.198328 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781938634772379-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (721 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 06:57:57.201917 388 mini_oidc.cc:152] Starting JWKS server
I20260620 06:57:57.203648 388 webserver.cc:460] Webserver started at https://127.0.0.1:33427/ using document root <none> and password file <none>
I20260620 06:57:57.203864 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:57:57.204453 388 webserver.cc:460] Webserver started at http://127.0.0.1:42723/ using document root <none> and password file <none>
2026-06-20T06:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:57Z Disabled control of system clock
I20260620 06:57:57.228518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39885
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39885
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/testchainca.cert with env {}
W20260620 06:57:57.543705 5667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:57.544010 5667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:57.544107 5667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:57.551432 5667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:57.551553 5667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:57.551589 5667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:57.551645 5667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:57.551688 5667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:57:57.551748 5667 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:57:57.551788 5667 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33427/jwks.json
I20260620 06:57:57.563267 5667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:57.564535 5667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:57.565788 5667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:57.577343 5672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:57.577378 5675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:57.577406 5673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:57.578145 5667 server_base.cc:1034] running on GCE node
I20260620 06:57:57.578998 5667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:57.580437 5667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:57.581640 5667 hybrid_clock.cc:648] HybridClock initialized: now 1781938677581587 us; error 48 us; skew 500 ppm
I20260620 06:57:57.582008 5667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:57.584194 5667 webserver.cc:460] Webserver started at http://127.0.97.62:41099/ using document root <none> and password file <none>
I20260620 06:57:57.584851 5667 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:57.584957 5667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:57.585261 5667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:57.588306 5667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "7cdbb89665074ef88e4679b86221a277"
format_stamp: "Formatted at 2026-06-20 06:57:57 on dist-test-slave-mz20"
server_key: "07b17c7f68c6310a08ce230a05d4e947"
server_key_iv: "df0de6f5caf14b32f542ce1f238a4d8c"
server_key_version: "encryptionkey@0"
I20260620 06:57:57.589098 5667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cdbb89665074ef88e4679b86221a277"
format_stamp: "Formatted at 2026-06-20 06:57:57 on dist-test-slave-mz20"
server_key: "07b17c7f68c6310a08ce230a05d4e947"
server_key_iv: "df0de6f5caf14b32f542ce1f238a4d8c"
server_key_version: "encryptionkey@0"
I20260620 06:57:57.594138 5667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:57:57.597718 5681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:57.599247 5667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:57:57.599418 5667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "7cdbb89665074ef88e4679b86221a277"
format_stamp: "Formatted at 2026-06-20 06:57:57 on dist-test-slave-mz20"
server_key: "07b17c7f68c6310a08ce230a05d4e947"
server_key_iv: "df0de6f5caf14b32f542ce1f238a4d8c"
server_key_version: "encryptionkey@0"
I20260620 06:57:57.599613 5667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:57.624094 5667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:57.624929 5667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:57.625188 5667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:57.642180 5667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39885
I20260620 06:57:57.642208 5732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39885 every 8 connection(s)
I20260620 06:57:57.643503 5667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:57.645634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5667
I20260620 06:57:57.645838 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:57.646423 388 external_mini_cluster.cc:1428] Setting key 2d9b565542ec1b2022e409202ffec36d
I20260620 06:57:57.647581 5733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:57.661661 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:57.648353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.97.62:39885)
0620 06:57:57.649372 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:57.649387 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:57.649444 (+ 57us) server_negotiation.cc:407] Connection header received
0620 06:57:57.651086 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:57.651112 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:57.651720 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:57.652158 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:57.652692 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.654725 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.656090 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.657087 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.657286 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:57.658445 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:57.658457 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:57.658478 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:57.659282 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:57.659388 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:57.659415 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:57.659545 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:57.659777 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 06:57:57.660149 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":316,"threads_started":1}
I20260620 06:57:57.662528 5733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277: Bootstrap starting.
I20260620 06:57:57.665550 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:57:57.665796 5733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:57.667007 5733 log.cc:824] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:57.669780 5733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277: No bootstrap required, opened a new log
I20260620 06:57:57.676663 5733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } }
I20260620 06:57:57.676999 5733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:57.677069 5733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdbb89665074ef88e4679b86221a277, State: Initialized, Role: FOLLOWER
I20260620 06:57:57.677582 5733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } }
I20260620 06:57:57.677773 5733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:57.677867 5733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:57.677989 5733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:57.680024 5733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } }
I20260620 06:57:57.680469 5733 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdbb89665074ef88e4679b86221a277; no voters:
I20260620 06:57:57.680945 5733 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:57.681098 5738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:57.682272 5738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [term 1 LEADER]: Becoming Leader. State: Replica: 7cdbb89665074ef88e4679b86221a277, State: Running, Role: LEADER
I20260620 06:57:57.683051 5738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } }
I20260620 06:57:57.683185 5733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:57.685653 5740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdbb89665074ef88e4679b86221a277. Latest consensus state: current_term: 1 leader_uuid: "7cdbb89665074ef88e4679b86221a277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } } }
I20260620 06:57:57.685956 5740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:57.686481 5739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdbb89665074ef88e4679b86221a277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdbb89665074ef88e4679b86221a277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39885 } } }
I20260620 06:57:57.686774 5739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:57.686842 5747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:57.692648 5747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:57.701638 5747 catalog_manager.cc:1347] Generated new cluster ID: b3011081cf284d6a910fa9d2fc4771c4
I20260620 06:57:57.701764 5747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:57.722889 5747 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:57.724735 5747 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:57.738052 5747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdbb89665074ef88e4679b86221a277: Generated new TSK 0
I20260620 06:57:57.739143 5747 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:57:57.776098 5764 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 06:57:57.776206 5764 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:57:57.777436 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:57.757310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.97.62:39885)
0620 06:57:57.757500 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:57.757506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:57.757871 (+ 365us) server_negotiation.cc:407] Connection header received
0620 06:57:57.758085 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:57.758090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:57.758179 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:57.758352 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:57:57.758812 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.759900 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.761196 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.761869 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.761986 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:57.762516 (+ 530us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:57:57.776738 (+ 14222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:57:57.777127 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 06:57:57.777510 5763 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:57:57.757156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39885 (local address 127.0.0.1:34402)
0620 06:57:57.757771 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:57.757787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:57.757913 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:57.758351 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:57.758354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:57.758362 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:57:57.758667 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:57.758677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.760016 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.760020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:57.761070 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:57.761081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:57.762001 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:57.762042 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:57.762195 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:57.762230 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:57:57.776983 (+ 14753us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:57:57.777185 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39885: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
W20260620 06:57:57.777558 5736 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 06:57:57.779541 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5667
I20260620 06:57:57.780716 5728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:57.875129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5667
2026-06-20T06:57:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (726 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 06:57:57.928504 388 mini_oidc.cc:152] Starting JWKS server
I20260620 06:57:57.930351 388 webserver.cc:460] Webserver started at https://127.0.0.1:38543/ using document root <none> and password file <none>
I20260620 06:57:57.930616 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:57:57.931252 388 webserver.cc:460] Webserver started at http://127.0.0.1:43897/ using document root <none> and password file <none>
2026-06-20T06:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:57Z Disabled control of system clock
I20260620 06:57:57.955560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45053
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:57:58.267592 5771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:58.267902 5771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:58.267992 5771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:58.275215 5771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:58.275406 5771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:58.275460 5771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:58.275516 5771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:58.275559 5771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:57:58.275619 5771 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:57:58.275656 5771 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38543/jwks.json
I20260620 06:57:58.284888 5771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:58.286140 5771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:58.287895 5771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:58.299374 5776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:58.299347 5777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:58.299296 5779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:58.299790 5771 server_base.cc:1034] running on GCE node
I20260620 06:57:58.300686 5771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:58.302095 5771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:58.303296 5771 hybrid_clock.cc:648] HybridClock initialized: now 1781938678303246 us; error 42 us; skew 500 ppm
I20260620 06:57:58.303633 5771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:58.308928 5771 webserver.cc:460] Webserver started at http://127.0.97.62:40097/ using document root <none> and password file <none>
I20260620 06:57:58.309613 5771 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:58.309707 5771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:58.310029 5771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:58.312961 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "fff66fbcc9f449f5a692d452acff8c15"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "ea41bf444564f785dfcd6f7b94bbf21d"
server_key_iv: "2eb72f1dc3697c1478609c14c55cd899"
server_key_version: "encryptionkey@0"
I20260620 06:57:58.313709 5771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "fff66fbcc9f449f5a692d452acff8c15"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "ea41bf444564f785dfcd6f7b94bbf21d"
server_key_iv: "2eb72f1dc3697c1478609c14c55cd899"
server_key_version: "encryptionkey@0"
I20260620 06:57:58.318954 5771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 06:57:58.322500 5785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:58.323861 5771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:57:58.324023 5771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "fff66fbcc9f449f5a692d452acff8c15"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "ea41bf444564f785dfcd6f7b94bbf21d"
server_key_iv: "2eb72f1dc3697c1478609c14c55cd899"
server_key_version: "encryptionkey@0"
I20260620 06:57:58.324227 5771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:58.355288 5771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:58.356196 5771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:58.356460 5771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:58.372747 5771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45053
I20260620 06:57:58.372766 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45053 every 8 connection(s)
I20260620 06:57:58.373987 5771 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:58.377578 5837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:58.382793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5771
I20260620 06:57:58.382980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:58.383715 388 external_mini_cluster.cc:1428] Setting key c06b956e6f4eddaff5e74551be91d837
I20260620 06:57:58.388401 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15: Bootstrap starting.
I20260620 06:57:58.391316 5837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:58.392249 5837 log.cc:824] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:58.395116 5837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15: No bootstrap required, opened a new log
I20260620 06:57:58.396389 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:58.385642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.97.62:45053)
0620 06:57:58.386636 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:58.386647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:58.386687 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:57:58.388293 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:58.388314 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:58.388774 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:58.389129 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:58.389659 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.390938 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.392216 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.392958 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.393133 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:58.394248 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:58.394256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:58.394270 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:58.394814 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:58.394893 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:58.394913 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:58.395014 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:58.395214 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 06:57:58.395499 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":310,"threads_started":1}
I20260620 06:57:58.399924 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:57:58.402532 5837 raft_consensus.cc:348] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260620 06:57:58.402930 5837 raft_consensus.cc:374] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:58.403024 5837 raft_consensus.cc:729] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff66fbcc9f449f5a692d452acff8c15, State: Initialized, Role: FOLLOWER
I20260620 06:57:58.403540 5837 consensus_queue.cc:260] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260620 06:57:58.403748 5837 raft_consensus.cc:388] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:58.403865 5837 raft_consensus.cc:482] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:58.404011 5837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:58.406152 5837 raft_consensus.cc:504] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260620 06:57:58.406668 5837 leader_election.cc:302] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fff66fbcc9f449f5a692d452acff8c15; no voters:
I20260620 06:57:58.407151 5837 leader_election.cc:288] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:58.407284 5842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:58.408401 5842 raft_consensus.cc:686] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [term 1 LEADER]: Becoming Leader. State: Replica: fff66fbcc9f449f5a692d452acff8c15, State: Running, Role: LEADER
I20260620 06:57:58.408857 5842 consensus_queue.cc:237] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } }
I20260620 06:57:58.409253 5837 sys_catalog.cc:564] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:58.411222 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fff66fbcc9f449f5a692d452acff8c15. Latest consensus state: current_term: 1 leader_uuid: "fff66fbcc9f449f5a692d452acff8c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260620 06:57:58.411510 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:58.411319 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fff66fbcc9f449f5a692d452acff8c15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fff66fbcc9f449f5a692d452acff8c15" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45053 } } }
I20260620 06:57:58.411680 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:58.412268 5849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:58.417642 5849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:58.428231 5849 catalog_manager.cc:1347] Generated new cluster ID: 59b8a4e4ce854323ae364d8188d70bbe
I20260620 06:57:58.428372 5849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:58.451776 5849 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:58.453339 5849 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:58.458666 5849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fff66fbcc9f449f5a692d452acff8c15: Generated new TSK 0
I20260620 06:57:58.459458 5849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:57:58.481882 5840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:58.474766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.0.97.62:45053)
0620 06:57:58.474984 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:58.474993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:58.475513 (+ 520us) server_negotiation.cc:407] Connection header received
0620 06:57:58.475739 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:58.475743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:58.475809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:58.475950 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:57:58.476428 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.477149 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.478479 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.478941 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.479064 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:58.479536 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:57:58.481284 (+ 1748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:57:58.481590 (+ 306us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 06:57:58.482110 5866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:57:58.474669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45053 (local address 127.0.0.1:51684)
0620 06:57:58.475415 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:58.475431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:58.475568 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:58.475989 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:58.475994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:58.476006 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:57:58.476281 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:58.476291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.477297 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.477302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:58.478328 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:58.478339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:58.479055 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:58.479131 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:58.479261 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:58.479294 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:57:58.481474 (+ 2180us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:57:58.481692 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45053: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":479,"threads_started":1}
W20260620 06:57:58.482419 5840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51684: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38543/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260620 06:57:58.482934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5771
I20260620 06:57:58.484151 5832 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 06:57:58.493770 5867 webserver.cc:533] Webserver: sslize failed:
E20260620 06:57:58.493875 5867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:57:58.574184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5771
2026-06-20T06:57:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (697 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T06:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:57:58Z Disabled control of system clock
I20260620 06:57:58.649710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44333
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44333
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/testchainca.cert with env {}
W20260620 06:57:58.957230 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:57:58.957655 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:57:58.957841 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:57:58.965106 5872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:57:58.965219 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:57:58.965265 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:57:58.965348 5872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:57:58.965410 5872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:57:58.965476 5872 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:57:58.965535 5872 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 06:57:58.975296 5872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44333
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:57:58.976554 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:57:58.977808 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:57:58.989207 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:57:58.989485 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:58.989741 5872 server_base.cc:1034] running on GCE node
W20260620 06:57:58.989288 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:57:58.990770 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:57:58.992146 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:57:58.993373 5872 hybrid_clock.cc:648] HybridClock initialized: now 1781938678993316 us; error 47 us; skew 500 ppm
I20260620 06:57:58.993754 5872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:57:58.995849 5872 webserver.cc:460] Webserver started at http://127.0.97.62:45755/ using document root <none> and password file <none>
I20260620 06:57:58.996507 5872 fs_manager.cc:359] Metadata directory not provided
I20260620 06:57:58.996606 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:57:58.996889 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:57:58.999662 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "adbd8781a82c4f55aa4577394d281d90"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "782e18b0effd6955e43b12a2209e181f"
server_key_iv: "32141597f202994105290a6197502817"
server_key_version: "encryptionkey@0"
I20260620 06:57:59.000397 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "adbd8781a82c4f55aa4577394d281d90"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "782e18b0effd6955e43b12a2209e181f"
server_key_iv: "32141597f202994105290a6197502817"
server_key_version: "encryptionkey@0"
I20260620 06:57:59.005244 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 06:57:59.008530 5886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:57:59.009797 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:57:59.009951 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "adbd8781a82c4f55aa4577394d281d90"
format_stamp: "Formatted at 2026-06-20 06:57:58 on dist-test-slave-mz20"
server_key: "782e18b0effd6955e43b12a2209e181f"
server_key_iv: "32141597f202994105290a6197502817"
server_key_version: "encryptionkey@0"
I20260620 06:57:59.010154 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:57:59.026909 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:57:59.027742 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:57:59.027998 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:57:59.044668 5872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44333
I20260620 06:57:59.044687 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44333 every 8 connection(s)
I20260620 06:57:59.045940 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:57:59.046691 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5872
I20260620 06:57:59.046886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:57:59.047590 388 external_mini_cluster.cc:1428] Setting key 5204329ac5d7437fce1138880ab43235
I20260620 06:57:59.050203 5938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:57:59.062865 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:59.049376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.0.97.62:44333)
0620 06:57:59.050388 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:59.050403 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:59.050453 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:57:59.052632 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:59.052657 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:59.053275 (+ 618us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:59.053746 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:57:59.054272 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.056255 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.057755 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.058760 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.058944 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:59.060040 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:57:59.060051 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:57:59.060071 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:57:59.060822 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:57:59.060913 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:57:59.060939 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:57:59.061056 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:57:59.061278 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 06:57:59.061605 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":295,"threads_started":1}
I20260620 06:57:59.063570 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: Bootstrap starting.
I20260620 06:57:59.066798 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:57:59.067111 5938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: Neither blocks nor log segments found. Creating new log.
I20260620 06:57:59.068148 5938 log.cc:824] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: Log is configured to *not* fsync() on all Append() calls
I20260620 06:57:59.071108 5938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: No bootstrap required, opened a new log
I20260620 06:57:59.078260 5938 raft_consensus.cc:348] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } }
I20260620 06:57:59.078649 5938 raft_consensus.cc:374] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:57:59.078751 5938 raft_consensus.cc:729] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd8781a82c4f55aa4577394d281d90, State: Initialized, Role: FOLLOWER
I20260620 06:57:59.079335 5938 consensus_queue.cc:260] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } }
I20260620 06:57:59.079604 5938 raft_consensus.cc:388] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:57:59.079708 5938 raft_consensus.cc:482] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:57:59.079829 5938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:57:59.081765 5938 raft_consensus.cc:504] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } }
I20260620 06:57:59.082207 5938 leader_election.cc:302] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adbd8781a82c4f55aa4577394d281d90; no voters:
I20260620 06:57:59.082772 5938 leader_election.cc:288] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:57:59.082890 5943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:57:59.083498 5943 raft_consensus.cc:686] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [term 1 LEADER]: Becoming Leader. State: Replica: adbd8781a82c4f55aa4577394d281d90, State: Running, Role: LEADER
I20260620 06:57:59.083990 5943 consensus_queue.cc:237] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } }
I20260620 06:57:59.084678 5938 sys_catalog.cc:564] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:57:59.086731 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adbd8781a82c4f55aa4577394d281d90. Latest consensus state: current_term: 1 leader_uuid: "adbd8781a82c4f55aa4577394d281d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } } }
I20260620 06:57:59.086802 5944 sys_catalog.cc:455] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adbd8781a82c4f55aa4577394d281d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adbd8781a82c4f55aa4577394d281d90" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44333 } } }
I20260620 06:57:59.087074 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [sys.catalog]: This master's current role is: LEADER
I20260620 06:57:59.087145 5944 sys_catalog.cc:458] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90 [sys.catalog]: This master's current role is: LEADER
W20260620 06:57:59.095340 5959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:57:59.095446 5959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:57:59.095563 5950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:57:59.096993 5950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:57:59.104876 5950 catalog_manager.cc:1347] Generated new cluster ID: c727b65e69ae4aec9b01b3dc7200f8bf
I20260620 06:57:59.104980 5950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:57:59.129773 5950 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:57:59.131618 5950 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:57:59.139716 5950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adbd8781a82c4f55aa4577394d281d90: Generated new TSK 0
I20260620 06:57:59.140739 5950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:57:59.625842 5941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:57:59.163158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.0.97.62:44333)
0620 06:57:59.163376 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 06:57:59.163383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:57:59.163759 (+ 376us) server_negotiation.cc:407] Connection header received
0620 06:57:59.164013 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:57:59.164017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:57:59.164107 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:57:59.164231 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:57:59.164742 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.165528 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.167521 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.167991 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.168092 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:59.168673 (+ 581us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:57:59.625259 (+456586us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:57:59.625603 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35230: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 06:57:59.625955 5941 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35230: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 06:57:59.626159 5967 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:57:59.163035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44333 (local address 127.0.0.1:35230)
0620 06:57:59.163643 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 06:57:59.163663 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:57:59.163819 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:57:59.164274 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:57:59.164279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:57:59.164293 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:57:59.164583 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:59.164594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.165672 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.165678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:59.167379 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:57:59.167395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:57:59.168109 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:57:59.168162 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:57:59.168349 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:57:59.168395 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:57:59.625533 (+457138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:57:59.625809 (+ 276us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44333: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
I20260620 06:57:59.627494 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5872
I20260620 06:57:59.628607 5933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:57:59.728277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5872
2026-06-20T06:57:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1135 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (404 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1395 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:01 dist-test-slave-mz20 krb5kdc[5983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:01 dist-test-slave-mz20 krb5kdc[5983](info): set up 2 sockets
Jun 20 06:58:01 dist-test-slave-mz20 krb5kdc[5983](info): commencing operation
krb5kdc: starting...
W20260620 06:58:02.530861 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:02 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:58:02.720320 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32821
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:03.033648 5996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:03.033934 5996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:03.034029 5996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:03.041285 5996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:03.041404 5996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:03.041463 5996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:03.041553 5996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:03.041606 5996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:03.051615 5996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:03.052907 5996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:03.054239 5996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:03.065747 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:03.065789 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:03.066200 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:03.066920 5996 server_base.cc:1034] running on GCE node
I20260620 06:58:03.067791 5996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:03.069202 5996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:03.070399 5996 hybrid_clock.cc:648] HybridClock initialized: now 1781938683070349 us; error 42 us; skew 500 ppm
I20260620 06:58:03.070770 5996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:03 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:03.073768 5996 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:58:03.075337 5996 webserver.cc:460] Webserver started at http://127.0.97.62:37281/ using document root <none> and password file <none>
I20260620 06:58:03.075999 5996 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:03.076104 5996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:03.076363 5996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:03.079317 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "ef514ad03cd74f50928bc1ceb326870d"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "d116b482d2366b8e5de7cdab5b2d5ea9"
server_key_iv: "b30531708f8b1bfceb511b2a703d2fd6"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.079999 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef514ad03cd74f50928bc1ceb326870d"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "d116b482d2366b8e5de7cdab5b2d5ea9"
server_key_iv: "b30531708f8b1bfceb511b2a703d2fd6"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.084946 5996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:58:03.088644 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:03.089933 5996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:03.090123 5996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "ef514ad03cd74f50928bc1ceb326870d"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "d116b482d2366b8e5de7cdab5b2d5ea9"
server_key_iv: "b30531708f8b1bfceb511b2a703d2fd6"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.090301 5996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:03.108613 5996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:03.112748 5996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:03.113022 5996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:03.130059 5996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32821
I20260620 06:58:03.130115 6062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32821 every 8 connection(s)
I20260620 06:58:03.131371 5996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:03.135136 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:03.138573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 5996
I20260620 06:58:03.138790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:03.139549 388 external_mini_cluster.cc:1428] Setting key fb3c9ea8f81c41a477cde78171077483
I20260620 06:58:03.147892 6063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d: Bootstrap starting.
I20260620 06:58:03.151563 6063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:03.152827 6063 log.cc:824] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:58:03 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:03.155987 6063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d: No bootstrap required, opened a new log
I20260620 06:58:03.161705 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:03.141534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.97.62:32821)
0620 06:58:03.142251 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:03.142263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:03.142304 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:58:03.143916 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:03.143939 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:03.144363 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:03.144690 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:03.146325 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.147702 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.149414 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.150112 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.150249 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:03.154829 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:03.154837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:03.154852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:03.154884 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:03.157766 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:03.158583 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:03.158592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:03.158603 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:03.158767 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:03.159118 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:03.159121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:03.159123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:03.159627 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:03.159828 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:03.160298 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 06:58:03.160661 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":329,"threads_started":1}
I20260620 06:58:03.164793 6063 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260620 06:58:03.165153 6063 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:03.165232 6063 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef514ad03cd74f50928bc1ceb326870d, State: Initialized, Role: FOLLOWER
I20260620 06:58:03.165784 6063 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260620 06:58:03.165987 6063 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:03.166105 6063 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:03.166244 6063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:03.168365 6063 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260620 06:58:03.168829 6063 leader_election.cc:302] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef514ad03cd74f50928bc1ceb326870d; no voters:
I20260620 06:58:03.169353 6063 leader_election.cc:288] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:03.169474 6068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:03.170616 6068 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [term 1 LEADER]: Becoming Leader. State: Replica: ef514ad03cd74f50928bc1ceb326870d, State: Running, Role: LEADER
I20260620 06:58:03.171166 6068 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } }
I20260620 06:58:03.171389 6063 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:03.173939 6070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef514ad03cd74f50928bc1ceb326870d. Latest consensus state: current_term: 1 leader_uuid: "ef514ad03cd74f50928bc1ceb326870d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } } }
I20260620 06:58:03.174253 6070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:03.174714 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef514ad03cd74f50928bc1ceb326870d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef514ad03cd74f50928bc1ceb326870d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32821 } } }
I20260620 06:58:03.174973 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:03.175693 6077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:03.180576 6077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:03.190117 6077 catalog_manager.cc:1347] Generated new cluster ID: 0a40d26d3e524717885e4bcdb0b14cb9
I20260620 06:58:03.190222 6077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:03.201941 6077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:03.203202 6077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:03.208798 6077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef514ad03cd74f50928bc1ceb326870d: Generated new TSK 0
I20260620 06:58:03.209595 6077 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:58:03.268046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:03.600571 6089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:03.600831 6089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:03.600920 6089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:03.607928 6089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:03.608039 6089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:03.608158 6089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:03.617765 6089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:03.618965 6089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:03.620206 6089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:03.632530 6094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:03.632592 6095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:03.632563 6097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:03.633767 6089 server_base.cc:1034] running on GCE node
I20260620 06:58:03.634392 6089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:03.635339 6089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:03.636562 6089 hybrid_clock.cc:648] HybridClock initialized: now 1781938683636487 us; error 65 us; skew 500 ppm
I20260620 06:58:03.636878 6089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:03 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:03.644260 6089 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:58:03.645772 6089 webserver.cc:460] Webserver started at http://127.0.97.1:44635/ using document root <none> and password file <none>
I20260620 06:58:03.646447 6089 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:03.646598 6089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:03.646889 6089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:03.649744 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "11afedfdb27f47adb90f255838448d07"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "c6f2fb44f78b93bb20ab62db081aa907"
server_key_iv: "1d5aa29eaf781bd235bf6596b3b31923"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.650470 6089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "11afedfdb27f47adb90f255838448d07"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "c6f2fb44f78b93bb20ab62db081aa907"
server_key_iv: "1d5aa29eaf781bd235bf6596b3b31923"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.655377 6089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260620 06:58:03.658670 6104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:03.660010 6089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:58:03.660233 6089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "11afedfdb27f47adb90f255838448d07"
format_stamp: "Formatted at 2026-06-20 06:58:03 on dist-test-slave-mz20"
server_key: "c6f2fb44f78b93bb20ab62db081aa907"
server_key_iv: "1d5aa29eaf781bd235bf6596b3b31923"
server_key_version: "encryptionkey@0"
I20260620 06:58:03.660429 6089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:03.686299 6089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:03.690485 6089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:03.690798 6089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:03.691712 6089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:03.693074 6089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:03.693150 6089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:03.693284 6089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:03.693336 6089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:03.721992 6089 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37235
I20260620 06:58:03.722045 6216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37235 every 8 connection(s)
I20260620 06:58:03.723217 6089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:03.729785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6089
I20260620 06:58:03.729987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:03.730684 388 external_mini_cluster.cc:1428] Setting key ecd8d16edda1b9910a8148f12230832d
Jun 20 06:58:03 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:03.741603 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:03.725548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53055 (local address 127.0.97.62:32821)
0620 06:58:03.725944 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:03.725949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:03.726821 (+ 872us) server_negotiation.cc:407] Connection header received
0620 06:58:03.728556 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:03.728561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:03.728650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:03.728746 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:03.730259 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.731043 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.732554 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.732936 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.733064 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:03.736369 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:03.736374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:03.736377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:03.736408 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:03.738785 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:03.739451 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:03.739455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:03.739458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:03.739569 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:03.739959 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:03.739964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:03.739967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:03.740228 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:03.740364 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:03.741036 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 06:58:03.741266 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":205,"threads_started":1}
I20260620 06:58:03.742771 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:03.725972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32821 (local address 127.0.97.1:53055)
0620 06:58:03.726668 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:03.726719 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:03.728296 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:03.728976 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:03.728985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:03.729489 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:03.730090 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:03.730115 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.731187 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.731191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:03.732414 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:03.732430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:03.733165 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:03.733172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:03.733402 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:03.734050 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:03.734071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:03.736161 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:03.738911 (+ 2750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:03.738920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:03.738938 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:03.739342 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:03.739665 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:03.739670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:03.739674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:03.739857 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:03.740346 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:03.740363 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:03.740678 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 06:58:03.740946 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 06:58:03.741467 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":258,"threads_started":1}
I20260620 06:58:03.744551 6217 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32821
I20260620 06:58:03.744863 6217 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:03.745585 6217 heartbeater.cc:511] Master 127.0.97.62:32821 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 06:58:03.747886 6028 ts_manager.cc:194] Registered new tserver with Master: 11afedfdb27f47adb90f255838448d07 (127.0.97.1:37235)
I20260620 06:58:03.749328 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:58:03.773034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:04.084079 6223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:04.084352 6223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:04.084442 6223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:04.091583 6223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:04.091688 6223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:04.091807 6223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:04.101531 6223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:04.102733 6223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:04.103960 6223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:04.116364 6228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:04.116396 6229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:04.116338 6231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:04.116991 6223 server_base.cc:1034] running on GCE node
I20260620 06:58:04.117604 6223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:04.118430 6223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:04.119617 6223 hybrid_clock.cc:648] HybridClock initialized: now 1781938684119564 us; error 44 us; skew 500 ppm
I20260620 06:58:04.119900 6223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.122879 6223 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:58:04.124238 6223 webserver.cc:460] Webserver started at http://127.0.97.2:43755/ using document root <none> and password file <none>
I20260620 06:58:04.124867 6223 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:04.124969 6223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:04.125254 6223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:04.128082 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "6c154abf2405467abf426c9c1fe4ddae"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "5b1af1f86355c4a6ad39587c64c67927"
server_key_iv: "28e822dc24c2755d444a8ed732fa58c7"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.128825 6223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6c154abf2405467abf426c9c1fe4ddae"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "5b1af1f86355c4a6ad39587c64c67927"
server_key_iv: "28e822dc24c2755d444a8ed732fa58c7"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.133740 6223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 06:58:04.137104 6238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.138509 6223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:04.138700 6223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "6c154abf2405467abf426c9c1fe4ddae"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "5b1af1f86355c4a6ad39587c64c67927"
server_key_iv: "28e822dc24c2755d444a8ed732fa58c7"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.138876 6223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:04.166311 6223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:04.170212 6223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:04.170464 6223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:04.171366 6223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:04.172648 6223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:04.172720 6223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.172832 6223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:04.172894 6223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.202476 6223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38465
I20260620 06:58:04.202522 6350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38465 every 8 connection(s)
I20260620 06:58:04.203693 6223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:04.213465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6223
I20260620 06:58:04.213627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:04.214133 388 external_mini_cluster.cc:1428] Setting key 7130dbd2497fee8c871372564eec530d
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:04.223783 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.206171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33909 (local address 127.0.97.62:32821)
0620 06:58:04.206341 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.206345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.207462 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 06:58:04.209295 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.209301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.209367 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.209454 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.211015 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.211778 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.213434 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.214027 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.214180 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.218195 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.218200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.218203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.218235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.220598 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.221425 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.221429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.221432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.221566 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.222074 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.222078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.222081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.222385 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.222501 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.223251 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.223474 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:58:04.224588 6353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.206626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32821 (local address 127.0.97.2:33909)
0620 06:58:04.207299 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:04.207341 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:04.209000 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:04.209696 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:04.209704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:04.210221 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:04.210856 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:04.210874 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.211927 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.211931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:04.213261 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:04.213282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.214347 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.214356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:04.214709 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.215399 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:04.215422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:04.217954 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:04.220730 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:04.220737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:04.220753 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:04.221281 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:04.221685 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:04.221690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:04.221693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:04.221936 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:04.222504 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:04.222513 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:04.222884 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 06:58:04.223105 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 06:58:04.223499 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260620 06:58:04.225970 6351 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32821
I20260620 06:58:04.226294 6351 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:04.227172 6351 heartbeater.cc:511] Master 127.0.97.62:32821 requested a full tablet report, sending...
I20260620 06:58:04.228649 6028 ts_manager.cc:194] Registered new tserver with Master: 6c154abf2405467abf426c9c1fe4ddae (127.0.97.2:38465)
I20260620 06:58:04.229629 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33909
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:58:04.257194 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:04.566288 6356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:04.566614 6356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:04.566771 6356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:04.574131 6356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:04.574249 6356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:04.574395 6356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:04.584064 6356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:04.585444 6356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:04.586771 6356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:04.599324 6361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:04.599350 6364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:04.599689 6362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:04.600364 6356 server_base.cc:1034] running on GCE node
I20260620 06:58:04.600966 6356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:04.601861 6356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:04.603053 6356 hybrid_clock.cc:648] HybridClock initialized: now 1781938684603003 us; error 40 us; skew 500 ppm
I20260620 06:58:04.603346 6356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.606151 6356 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:58:04.607559 6356 webserver.cc:460] Webserver started at http://127.0.97.3:33277/ using document root <none> and password file <none>
I20260620 06:58:04.608232 6356 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:04.608345 6356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:04.608614 6356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:04.611593 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "8d347df7333e427bb3ed79874d9071c8"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "c33c09518dfb9406a58a126d0a24b45c"
server_key_iv: "0f7ef57f9bf849162a0a9ce20a9996de"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.612342 6356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8d347df7333e427bb3ed79874d9071c8"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "c33c09518dfb9406a58a126d0a24b45c"
server_key_iv: "0f7ef57f9bf849162a0a9ce20a9996de"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.617427 6356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:58:04.620855 6371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.622164 6356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:04.622318 6356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "8d347df7333e427bb3ed79874d9071c8"
format_stamp: "Formatted at 2026-06-20 06:58:04 on dist-test-slave-mz20"
server_key: "c33c09518dfb9406a58a126d0a24b45c"
server_key_iv: "0f7ef57f9bf849162a0a9ce20a9996de"
server_key_version: "encryptionkey@0"
I20260620 06:58:04.622496 6356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:04.635808 6356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:04.640203 6356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:04.640497 6356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:04.641491 6356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:04.642889 6356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:04.642966 6356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.643082 6356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:04.643136 6356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:04.672698 6356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38111
I20260620 06:58:04.672762 6483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38111 every 8 connection(s)
I20260620 06:58:04.673969 6356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:04.675307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6356
I20260620 06:58:04.675455 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:04.675833 388 external_mini_cluster.cc:1428] Setting key e916237ba7d1be2c8fa03847200e9e76
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:04.694563 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.677922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44713 (local address 127.0.97.62:32821)
0620 06:58:04.678088 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.678092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.679232 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 06:58:04.681051 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.681055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.681116 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.681195 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.682587 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.683342 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.684788 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.685159 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.685246 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.688396 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.688401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.688404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.688435 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.691156 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.692078 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.692084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.692088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.692270 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.692738 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.692743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.692745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.693069 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.693225 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.694015 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.694247 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:58:04.695531 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.678398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32821 (local address 127.0.97.3:44713)
0620 06:58:04.679085 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:04.679119 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:04.680820 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:04.681371 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:04.681379 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:04.681869 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:04.682404 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:04.682420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.683463 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.683466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:04.684679 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:04.684690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.685347 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.685353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:04.685538 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.686137 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:04.686158 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:04.688193 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:04.691328 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:04.691335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:04.691349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:04.691902 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:04.692397 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:04.692400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:04.692403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:04.692596 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:04.693226 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:04.693233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:04.693632 (+ 399us) client_negotiation.cc:769] Sending connection context
0620 06:58:04.693893 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 06:58:04.694357 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":768,"threads_started":1}
I20260620 06:58:04.697006 6484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32821
I20260620 06:58:04.697314 6484 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:04.698067 6484 heartbeater.cc:511] Master 127.0.97.62:32821 requested a full tablet report, sending...
I20260620 06:58:04.699568 6028 ts_manager.cc:194] Registered new tserver with Master: 8d347df7333e427bb3ed79874d9071c8 (127.0.97.3:38111)
I20260620 06:58:04.700383 6028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44713
I20260620 06:58:04.705077 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/test-admin.keytab.
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.731256 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 06:58:04.736040 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:04.733677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.97.1:37235)
0620 06:58:04.734249 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.734258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.734290 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:58:04.734358 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.734367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.734605 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.734752 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.735633 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47006: BlockingRecv error: recv got EOF from 127.0.0.1:47006 (error 108)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":262,"threads_started":1}
W20260620 06:58:04.736151 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.737645 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.739005 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:58:04.752779 6217 heartbeater.cc:503] Master 127.0.97.62:32821 was elected leader, sending a full tablet report...
I20260620 06:58:04.753355 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.740580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.0.97.1:37235)
0620 06:58:04.740767 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.740773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.740917 (+ 144us) server_negotiation.cc:407] Connection header received
0620 06:58:04.741046 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.741049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.741105 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.741182 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.742131 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.743157 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.744322 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.744874 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.744974 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.748255 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.748265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.748279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.748322 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.750864 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.751458 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.751465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.751467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.751588 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.751913 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.751917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.751919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.752272 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.752483 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.752786 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.753004 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 06:58:04.758714 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:04.757108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.0.97.1:37235)
0620 06:58:04.757271 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.757277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.757556 (+ 279us) server_negotiation.cc:407] Connection header received
0620 06:58:04.757691 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.757696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.757764 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.757836 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.758470 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47024: BlockingRecv error: recv got EOF from 127.0.0.1:47024 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 06:58:04.758827 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.760200 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.761408 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:58:04.774439 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.762598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47034 (local address 127.0.97.1:37235)
0620 06:58:04.762738 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.762742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.763035 (+ 293us) server_negotiation.cc:407] Connection header received
0620 06:58:04.763148 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.763151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.763204 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.763278 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.764111 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.764941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.766048 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.766452 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.766578 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.769786 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.769791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.769794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.769824 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.772105 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.772723 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.772729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.772733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.772852 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.773181 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.773186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.773190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.773451 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.773697 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.773967 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.774189 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:58:04.778121 6491 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:04.776699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.0.97.1:37235)
0620 06:58:04.776884 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.776889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.777090 (+ 201us) server_negotiation.cc:407] Connection header received
0620 06:58:04.777198 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.777201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.777263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.777344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.777909 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47048: BlockingRecv error: recv got EOF from 127.0.0.1:47048 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:58:04.778263 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781938634772379-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.779609 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:04.780750 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:58:04.794488 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.781982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.0.97.1:37235)
0620 06:58:04.782134 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.782138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.782493 (+ 355us) server_negotiation.cc:407] Connection header received
0620 06:58:04.782605 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.782609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.782673 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.782761 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.783707 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.784431 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.785637 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.786060 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.786150 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.789408 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.789412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.789416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.789447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.792089 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.792710 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.792713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.792715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.792857 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.793215 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.793217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.793219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.793470 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.793718 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.793946 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.794158 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:58:04.795852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6089
I20260620 06:58:04.807101 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.796628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.0.97.1:37235)
0620 06:58:04.796799 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.796804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.796993 (+ 189us) server_negotiation.cc:407] Connection header received
0620 06:58:04.797138 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.797143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.797194 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.797269 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.798169 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.798893 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.800254 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.800638 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.800750 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.802626 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.802631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.802634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.802661 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.804824 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.805387 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.805392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.805396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.805508 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.805822 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.805827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.805830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.806062 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.806314 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.806625 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.806813 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:58:04.807682 6212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:04.890156 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6089
I20260620 06:58:04.904938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6223
Jun 20 06:58:04 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:58:04.919699 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:04.905846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.0.97.2:38465)
0620 06:58:04.906371 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:04.906378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:04.906404 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:58:04.906461 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:04.906468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:04.906727 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:04.906877 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:04.907754 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.908747 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.909913 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:04.910446 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:04.910571 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:04.914278 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:04.914290 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:04.914305 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:04.914339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:04.917009 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.917683 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.917690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.917693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.917819 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:04.918171 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:04.918175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:04.918177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:04.918517 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:04.918837 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:04.919136 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 06:58:04.919379 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260620 06:58:04.920954 6346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:05.001662 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6223
I20260620 06:58:05.013890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6356
Jun 20 06:58:05 dist-test-slave-mz20 krb5kdc[5983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:58:05.029244 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:05.015010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.0.97.3:38111)
0620 06:58:05.015596 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:05.015604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:05.015632 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:58:05.015691 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:05.015713 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:05.015922 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:05.016048 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:05.017008 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:05.018041 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:05.019348 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:05.019897 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:05.019990 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:05.023781 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:05.023792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:05.023808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:05.023861 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:05.026728 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:05.027373 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:05.027379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:05.027381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:05.027493 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:05.027886 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:05.027889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:05.027891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:05.028219 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:05.028468 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:05.028752 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 06:58:05.028959 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":304,"threads_started":1}
I20260620 06:58:05.030532 6479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:05.112534 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6356
I20260620 06:58:05.125391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 5996
I20260620 06:58:05.126559 6058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:05.227113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 5996
2026-06-20T06:58:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3701 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:05 dist-test-slave-mz20 krb5kdc[6511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:05 dist-test-slave-mz20 krb5kdc[6511](info): set up 2 sockets
Jun 20 06:58:05 dist-test-slave-mz20 krb5kdc[6511](info): commencing operation
krb5kdc: starting...
W20260620 06:58:06.231079 388 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:06 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 06:58:06.425536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37021
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:06.731858 6524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:06.732133 6524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:06.732244 6524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:06.739326 6524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:06.739428 6524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:06.739506 6524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:06.739579 6524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:06.739639 6524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:06.749161 6524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:06.750394 6524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:06.751673 6524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:06.762310 6529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:06.762331 6530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:06.762583 6532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:06.763003 6524 server_base.cc:1034] running on GCE node
I20260620 06:58:06.763679 6524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:06.765087 6524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:06.766283 6524 hybrid_clock.cc:648] HybridClock initialized: now 1781938686766233 us; error 45 us; skew 500 ppm
I20260620 06:58:06.766630 6524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:06 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:06.769702 6524 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 06:58:06.771165 6524 webserver.cc:460] Webserver started at http://127.0.97.62:42735/ using document root <none> and password file <none>
I20260620 06:58:06.771821 6524 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:06.771912 6524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:06.772161 6524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:06.775094 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "9cfb2df0979b4700aabe071cf98f4838"
format_stamp: "Formatted at 2026-06-20 06:58:06 on dist-test-slave-mz20"
server_key: "f7bafbd590dd320cb6ca6f53a68c5564"
server_key_iv: "6806b964bf2de70e1ccd26f79da953b5"
server_key_version: "encryptionkey@0"
I20260620 06:58:06.775799 6524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "9cfb2df0979b4700aabe071cf98f4838"
format_stamp: "Formatted at 2026-06-20 06:58:06 on dist-test-slave-mz20"
server_key: "f7bafbd590dd320cb6ca6f53a68c5564"
server_key_iv: "6806b964bf2de70e1ccd26f79da953b5"
server_key_version: "encryptionkey@0"
I20260620 06:58:06.780541 6524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:58:06.783903 6539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:06.785315 6524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:06.785499 6524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "9cfb2df0979b4700aabe071cf98f4838"
format_stamp: "Formatted at 2026-06-20 06:58:06 on dist-test-slave-mz20"
server_key: "f7bafbd590dd320cb6ca6f53a68c5564"
server_key_iv: "6806b964bf2de70e1ccd26f79da953b5"
server_key_version: "encryptionkey@0"
I20260620 06:58:06.785681 6524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:06.832504 6524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:06.836504 6524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:06.836760 6524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:06.853714 6524 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37021
I20260620 06:58:06.853730 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37021 every 8 connection(s)
I20260620 06:58:06.855109 6524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:06.858731 6591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:06.863716 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6524
I20260620 06:58:06.863930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:06.864653 388 external_mini_cluster.cc:1428] Setting key dd90d1ffbaf718269ce045798ca67f4e
I20260620 06:58:06.869495 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Bootstrap starting.
I20260620 06:58:06.872870 6591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:06.874094 6591 log.cc:824] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:06.877266 6591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: No bootstrap required, opened a new log
Jun 20 06:58:06 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:06.884963 6591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260620 06:58:06.885353 6591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:06.885448 6591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfb2df0979b4700aabe071cf98f4838, State: Initialized, Role: FOLLOWER
I20260620 06:58:06.885987 6591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260620 06:58:06.886207 6591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:06.886336 6591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:06.886483 6591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:06.886958 6594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:06.866744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.0.97.62:37021)
0620 06:58:06.867494 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:06.867510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:06.867554 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:58:06.869141 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:06.869166 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:06.869531 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:06.869855 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:06.871605 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:06.872849 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:06.874641 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:06.875609 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:06.875738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:06.879868 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:06.879875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:06.879890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:06.879931 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:06.882860 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:06.883485 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:06.883491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:06.883497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:06.883670 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:06.884039 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:06.884043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:06.884045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:06.884560 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:06.884820 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:06.885233 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 06:58:06.885831 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":357,"threads_started":1}
I20260620 06:58:06.889130 6591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260620 06:58:06.889653 6591 leader_election.cc:302] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cfb2df0979b4700aabe071cf98f4838; no voters:
I20260620 06:58:06.890194 6591 leader_election.cc:288] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:06.890321 6596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:06.890918 6596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 LEADER]: Becoming Leader. State: Replica: 9cfb2df0979b4700aabe071cf98f4838, State: Running, Role: LEADER
I20260620 06:58:06.891438 6596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } }
I20260620 06:58:06.892099 6591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:06.893754 6598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfb2df0979b4700aabe071cf98f4838. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } }
I20260620 06:58:06.894136 6598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:06.894577 6597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } }
I20260620 06:58:06.894965 6597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:06.895238 6604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:06.906091 6604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:06.915190 6604 catalog_manager.cc:1347] Generated new cluster ID: afdf28a77a0a41679c5939805be3bde4
I20260620 06:58:06.915293 6604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:06.932291 6604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:06.933610 6604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:06.957370 6604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Generated new TSK 0
I20260620 06:58:06.958256 6604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 06:58:07.013659 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:07.349308 6617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:07.349592 6617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:07.349709 6617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:07.356827 6617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:07.356933 6617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:07.357087 6617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:07.366676 6617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:07.367878 6617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:07.369128 6617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:07.381816 6622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:07.381819 6623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:07.381805 6625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:07.382386 6617 server_base.cc:1034] running on GCE node
I20260620 06:58:07.382964 6617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:07.383855 6617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:07.385039 6617 hybrid_clock.cc:648] HybridClock initialized: now 1781938687384982 us; error 53 us; skew 500 ppm
I20260620 06:58:07.385315 6617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:07 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:07.388386 6617 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 06:58:07.389808 6617 webserver.cc:460] Webserver started at http://127.0.97.1:34891/ using document root <none> and password file <none>
I20260620 06:58:07.390512 6617 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:07.390663 6617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:07.390928 6617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:07.393893 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "2e4d6d1c44f04db5ab57c8ca75c360de"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "2ca8fb86ecd929dc1fe036acb941e663"
server_key_iv: "e5abbce2fb85caac617838f89f19c15c"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.394724 6617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2e4d6d1c44f04db5ab57c8ca75c360de"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "2ca8fb86ecd929dc1fe036acb941e663"
server_key_iv: "e5abbce2fb85caac617838f89f19c15c"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.399796 6617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:58:07.403138 6632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.404420 6617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:07.404565 6617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "2e4d6d1c44f04db5ab57c8ca75c360de"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "2ca8fb86ecd929dc1fe036acb941e663"
server_key_iv: "e5abbce2fb85caac617838f89f19c15c"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.404759 6617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:07.446774 6617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:07.450443 6617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:07.450736 6617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:07.451723 6617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:07.452982 6617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:07.453056 6617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.453173 6617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:07.453225 6617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.485350 6617 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41147
I20260620 06:58:07.485421 6744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41147 every 8 connection(s)
I20260620 06:58:07.486662 6617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:07.495826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6617
I20260620 06:58:07.496047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:07.496817 388 external_mini_cluster.cc:1428] Setting key 0682d1acc6f303f635ca1c86936bcc49
Jun 20 06:58:07 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:07.506160 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:07.489077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52895 (local address 127.0.97.62:37021)
0620 06:58:07.489508 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:07.489514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:07.490374 (+ 860us) server_negotiation.cc:407] Connection header received
0620 06:58:07.492253 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:07.492258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:07.492340 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:07.492443 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:07.494066 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:07.494847 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:07.496959 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:07.497384 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:07.497504 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:07.500896 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:07.500901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:07.500904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:07.500935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:07.503310 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:07.504062 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:07.504067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:07.504071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:07.504207 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:07.504554 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:07.504558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:07.504561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:07.504877 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:07.505000 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:07.505596 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 06:58:07.505832 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":204,"threads_started":1}
I20260620 06:58:07.506976 6748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:07.489549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:52895)
0620 06:58:07.490222 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:07.490265 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:07.491986 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:07.492736 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:07.492745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:07.493293 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:07.493883 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:07.493901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:07.495004 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:07.495008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:07.496808 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:07.496823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:07.497592 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:07.497600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:07.497822 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:07.498408 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:07.498430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:07.500692 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:07.503434 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:07.503443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:07.503460 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:07.503959 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:07.504300 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:07.504303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:07.504306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:07.504469 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:07.504977 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:07.504983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:07.505275 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 06:58:07.505490 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 06:58:07.505853 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":252,"threads_started":1}
I20260620 06:58:07.508373 6745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260620 06:58:07.508653 6745 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:07.509349 6745 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:07.511468 6556 ts_manager.cc:194] Registered new tserver with Master: 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:07.512923 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52895
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 06:58:07.539436 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:07.850957 6751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:07.851231 6751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:07.851366 6751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:07.861524 6751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:07.861690 6751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:07.861874 6751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:07.876662 6751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:07.877918 6751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:07.879231 6751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:07.891851 6756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:07.891819 6759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:07.892393 6757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:07.893265 6751 server_base.cc:1034] running on GCE node
I20260620 06:58:07.893925 6751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:07.894922 6751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:07.896122 6751 hybrid_clock.cc:648] HybridClock initialized: now 1781938687896086 us; error 35 us; skew 500 ppm
I20260620 06:58:07.896428 6751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:07 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:07.899412 6751 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 06:58:07.900800 6751 webserver.cc:460] Webserver started at http://127.0.97.2:35419/ using document root <none> and password file <none>
I20260620 06:58:07.901448 6751 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:07.901556 6751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:07.901860 6751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:07.904663 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "d389b7db88f141fe9b406b6197844bdc"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "116f66366dea78ac9c9995f7741438e5"
server_key_iv: "e1019b651daf345baae18631c7f55e0b"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.905396 6751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d389b7db88f141fe9b406b6197844bdc"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "116f66366dea78ac9c9995f7741438e5"
server_key_iv: "e1019b651daf345baae18631c7f55e0b"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.910292 6751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 06:58:07.913597 6766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.915011 6751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:07.915160 6751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "d389b7db88f141fe9b406b6197844bdc"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "116f66366dea78ac9c9995f7741438e5"
server_key_iv: "e1019b651daf345baae18631c7f55e0b"
server_key_version: "encryptionkey@0"
I20260620 06:58:07.915349 6751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:07.957007 6751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:07.961299 6751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:07.961560 6751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:07.962467 6751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:07.963748 6751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:07.963824 6751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.963936 6751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:07.963989 6751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:07.993220 6751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46257
I20260620 06:58:07.993287 6878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46257 every 8 connection(s)
I20260620 06:58:07.994422 6751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:07.997994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6751
I20260620 06:58:07.998200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:07.998746 388 external_mini_cluster.cc:1428] Setting key 3b454c1c47c05286b6b3bfdd5e3e12cf
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:08.014565 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:07.996841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45855 (local address 127.0.97.62:37021)
0620 06:58:07.996999 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:07.997003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:07.998296 (+ 1293us) server_negotiation.cc:407] Connection header received
0620 06:58:08.000317 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:08.000323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:08.000401 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:08.000486 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:08.001912 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.002653 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.004144 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.004551 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.004657 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:08.008278 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:08.008283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:08.008286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:08.008318 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:08.011344 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:08.012219 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:08.012224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:08.012228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:08.012372 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:08.012790 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:08.012795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:08.012798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:08.013120 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:08.013263 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:08.013949 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 06:58:08.014177 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:58:08.015379 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:07.997321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:45855)
0620 06:58:07.998127 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:07.998186 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:08.000071 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:08.000718 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:08.000727 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:08.001207 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:08.001751 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.001769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.002797 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.002800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:08.004007 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.004018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.004756 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.004763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:08.004970 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:08.005571 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:08.005592 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:08.008061 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:08.011510 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:08.011519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:08.011538 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:08.012113 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:08.012468 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:08.012471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:08.012473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:08.012695 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:08.013240 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:08.013246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:08.013554 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 06:58:08.013827 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 06:58:08.014215 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":323,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 06:58:08.016876 6879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260620 06:58:08.017195 6879 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:08.017886 6879 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:08.019464 6556 ts_manager.cc:194] Registered new tserver with Master: d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260620 06:58:08.020529 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 06:58:08.042192 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:08.354353 6884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:08.354772 6884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:08.354930 6884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:08.362210 6884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:08.362305 6884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:08.362416 6884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:08.372236 6884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:08.373471 6884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:08.374926 6884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:08.387398 6890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:08.387482 6892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:08.387537 6889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:08.387960 6884 server_base.cc:1034] running on GCE node
I20260620 06:58:08.388536 6884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:08.389463 6884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:08.390620 6884 hybrid_clock.cc:648] HybridClock initialized: now 1781938688390569 us; error 42 us; skew 500 ppm
I20260620 06:58:08.390959 6884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:08.393868 6884 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 06:58:08.395185 6884 webserver.cc:460] Webserver started at http://127.0.97.3:44065/ using document root <none> and password file <none>
I20260620 06:58:08.395820 6884 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:08.395929 6884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:08.396193 6884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:08.399089 6884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "bd85221f3ac548a29967a47c0a955ee5"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "b5f49f9029be75e58d02e218ba5bdf63"
server_key_iv: "643b5dfdeaa264b92863362c814ba8a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.399796 6884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bd85221f3ac548a29967a47c0a955ee5"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "b5f49f9029be75e58d02e218ba5bdf63"
server_key_iv: "643b5dfdeaa264b92863362c814ba8a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.404425 6884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:58:08.407557 6899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:08.408957 6884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:08.409098 6884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "bd85221f3ac548a29967a47c0a955ee5"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "b5f49f9029be75e58d02e218ba5bdf63"
server_key_iv: "643b5dfdeaa264b92863362c814ba8a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.409289 6884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:08.446360 6884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:08.450666 6884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:08.450925 6884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:08.451866 6884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:08.453147 6884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:08.453229 6884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:08.453356 6884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:08.453408 6884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:08.482759 6884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35001
I20260620 06:58:08.482823 7011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35001 every 8 connection(s)
I20260620 06:58:08.483960 6884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:08.490708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 6884
I20260620 06:58:08.490886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:08.491289 388 external_mini_cluster.cc:1428] Setting key 9fdeb5ba03945fcfa728c8329071f549
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:08.502971 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:08.486649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53763 (local address 127.0.97.62:37021)
0620 06:58:08.486894 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:08.486900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:08.488104 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 06:58:08.490052 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:08.490056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:08.490119 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:08.490209 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:08.491792 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.492756 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.494507 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.494919 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.495017 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:08.498185 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:08.498190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:08.498193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:08.498226 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:08.500494 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:08.501159 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:08.501162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:08.501164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:08.501278 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:08.501611 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:08.501614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:08.501616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:08.501858 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:08.501956 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:08.502533 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 06:58:08.502724 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:58:08.503897 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:08.487182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:53763)
0620 06:58:08.487934 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:08.487978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:08.489802 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:08.490420 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:08.490428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:08.491032 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:08.491599 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.491642 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.492891 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.492895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:08.494380 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.494392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.495124 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.495130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:08.495333 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:08.495937 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:08.495963 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:08.498021 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:08.500619 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:08.500625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:08.500637 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:08.501070 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:08.501355 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:08.501358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:08.501360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:08.501535 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:08.501946 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:08.501951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:08.502245 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 06:58:08.502460 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 06:58:08.502865 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":294,"threads_started":1}
I20260620 06:58:08.505221 7012 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260620 06:58:08.505519 7012 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:08.506302 7012 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:08.507570 6556 ts_manager.cc:194] Registered new tserver with Master: bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:08.508440 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53763
I20260620 06:58:08.516016 6745 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
I20260620 06:58:08.518201 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 06:58:08.559899 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:32775
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37021,127.0.97.61:32775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:08.876140 7017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:08.876410 7017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:08.876495 7017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:08.883530 7017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:08.883631 7017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:08.883682 7017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:08.883735 7017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:08.883776 7017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:08.893604 7017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37021,127.0.97.61:32775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:32775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
W20260620 06:58:08.893942 7017 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:37021,127.0.97.61:32775'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 06:58:08.894989 7017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:08.896250 7017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:08.907478 7022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:08.907670 7025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:08.907965 7023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:08.908464 7017 server_base.cc:1034] running on GCE node
I20260620 06:58:08.909154 7017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:08.910472 7017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:08.911722 7017 hybrid_clock.cc:648] HybridClock initialized: now 1781938688911666 us; error 28 us; skew 500 ppm
I20260620 06:58:08.912050 7017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:08.918116 7017 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 06:58:08.919646 7017 webserver.cc:460] Webserver started at http://127.0.97.61:36737/ using document root <none> and password file <none>
I20260620 06:58:08.920327 7017 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:08.920437 7017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:08.920718 7017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:08.923662 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/instance:
uuid: "9feaf91cbec040869784ceec0cbab59f"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "4bf7761b98eb73e4cb7c3d0bf050410f"
server_key_iv: "41acfabfe730af6cb5ec58b16247c638"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.924387 7017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal/instance:
uuid: "9feaf91cbec040869784ceec0cbab59f"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "4bf7761b98eb73e4cb7c3d0bf050410f"
server_key_iv: "41acfabfe730af6cb5ec58b16247c638"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.929425 7017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:58:08.932982 7032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:08.934458 7017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:58:08.934654 7017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "9feaf91cbec040869784ceec0cbab59f"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "4bf7761b98eb73e4cb7c3d0bf050410f"
server_key_iv: "41acfabfe730af6cb5ec58b16247c638"
server_key_version: "encryptionkey@0"
I20260620 06:58:08.934886 7017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:08.958319 7017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:08.962507 7017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:08.962774 7017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:08.979372 7017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:32775
I20260620 06:58:08.979391 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:32775 every 8 connection(s)
I20260620 06:58:08.980631 7017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
I20260620 06:58:08.984344 7084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:08.989012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7017
I20260620 06:58:08.991318 7084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 06:58:08 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 06:58:09.009464 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:08.993903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.0.97.62:37021)
0620 06:58:08.994069 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:08.994074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:08.994813 (+ 739us) server_negotiation.cc:407] Connection header received
0620 06:58:08.995147 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:08.995154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:08.995240 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:08.995325 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:08.997025 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.997726 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.999546 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.999959 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.000092 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.004007 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.004012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.004015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.004047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.006503 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.007225 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.007229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.007231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.007343 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.007794 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.007799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.007801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.008086 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.008231 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.008953 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.009158 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:58:09.010512 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:08.991203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.0.97.61:32775)
0620 06:58:08.991974 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:08.991986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:08.992026 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:58:08.994325 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:08.994362 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:08.994904 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:08.995178 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:08.996865 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.998621 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.000020 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.000681 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.000799 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.005065 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.005073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.005084 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.005110 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.007619 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.008202 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.008208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.008210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.008329 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.008715 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.008720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.008722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.009004 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.009278 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.009837 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.010022 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":562,"thread_start_us":327,"threads_started":1}
I20260620 06:58:09.010567 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:08.993920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39182)
0620 06:58:08.994669 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:08.994723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:08.994922 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:08.995536 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:08.995542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:08.996333 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:08.996853 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.996871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:08.997834 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:08.997839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:08.999403 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:08.999416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.000166 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.000171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.000383 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.001116 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.001145 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.003834 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.006629 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.006640 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.006651 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.007109 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.007445 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.007450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.007454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.007672 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.008201 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.008209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.008536 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.008820 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.009352 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":2,"thread_start_us":352,"threads_started":1}
I20260620 06:58:09.012231 7084 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 06:58:09.023025 6879 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
I20260620 06:58:09.024003 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.012963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50544)
0620 06:58:09.013159 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.013176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.013253 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.013522 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.013526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.013871 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.013989 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.013998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.014940 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.014944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.015819 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.015830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.016507 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.016511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.016659 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.017148 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.017163 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.019032 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.021669 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.021673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.021676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.022102 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.022449 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.022452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.022455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.022629 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.023118 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.023122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.023270 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.023409 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.023656 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 06:58:09.024092 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.012927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.0.97.61:32775)
0620 06:58:09.013084 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.013089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.013235 (+ 146us) server_negotiation.cc:407] Connection header received
0620 06:58:09.013337 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.013341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.013404 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.013492 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.014107 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.014826 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.015987 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.016375 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.016479 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.019184 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.019188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.019191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.019219 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.021527 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.022199 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.022202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.022205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.022336 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.022728 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.022732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.022735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.023004 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.023119 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.023493 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.023805 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":44}
I20260620 06:58:09.028610 7084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Bootstrap starting.
I20260620 06:58:09.031569 7084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:09.032488 7084 log.cc:824] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:09.035102 7084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: No bootstrap required, opened a new log
I20260620 06:58:09.042199 7084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } }
I20260620 06:58:09.042503 7084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:09.042608 7084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9feaf91cbec040869784ceec0cbab59f, State: Initialized, Role: FOLLOWER
I20260620 06:58:09.043128 7084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } }
I20260620 06:58:09.043850 7090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } } }
I20260620 06:58:09.044174 7090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:09.044806 7084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:58:09.052928 7101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:58:09.053076 7101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:58:09.055279 7017 master_runner.cc:428] Detected that this master 9feaf91cbec040869784ceec0cbab59f is joining an existing cluster
I20260620 06:58:09.055371 7017 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:32775
I20260620 06:58:09.074817 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.059385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50550)
0620 06:58:09.059997 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.060016 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.060154 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.060505 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.060510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.061161 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.061555 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.061568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.063058 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.063063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.064438 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.064448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.065595 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.065601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.065770 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.066402 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.066425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.067542 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.071541 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.071545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.071550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.072294 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.072658 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.072662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.072665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.072808 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.073724 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.073729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.073877 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.074030 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.074320 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":342,"threads_started":1}
I20260620 06:58:09.074863 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.059318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.0.97.61:32775)
0620 06:58:09.059508 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.059514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.060109 (+ 595us) server_negotiation.cc:407] Connection header received
0620 06:58:09.060288 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.060293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.060374 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.060473 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.061695 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.062900 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.064549 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.065274 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.065374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.067722 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.067728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.067733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.067765 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.071420 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.072423 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.072429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.072432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.072552 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.073144 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.073148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.073151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.073522 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.073657 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.074130 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.074500 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":84}
I20260620 06:58:09.077169 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.059355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.0.97.62:37021)
0620 06:58:09.059508 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.059512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.063077 (+ 3565us) server_negotiation.cc:407] Connection header received
0620 06:58:09.063234 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.063239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.063313 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.063402 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.065189 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.066181 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.068164 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.068777 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.068871 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.071160 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.071167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.071172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.071211 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.074429 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.075146 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.075152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.075155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.075316 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.075714 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.075719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.075722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.076068 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.076193 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.076597 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.076803 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:58:09.077235 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.059779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39190)
0620 06:58:09.062970 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.062988 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.063106 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.063438 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.063442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.064715 (+ 1273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.065016 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.065030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.066391 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.066396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.068013 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.068025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.069142 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.069147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.069330 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.069841 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.069864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.070959 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.074588 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.074593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.074597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.075016 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.075443 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.075448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.075452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.075593 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.076175 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.076180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.076347 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.076461 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.076866 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3058,"thread_start_us":2900,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 06:58:09.103945 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.082684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.0.97.61:32775)
0620 06:58:09.082842 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.082848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.087208 (+ 4360us) server_negotiation.cc:407] Connection header received
0620 06:58:09.087516 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.087520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.087599 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.087684 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.089182 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.090253 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.092412 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.093034 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.093166 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.097176 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.097183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.097188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.097227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.100676 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.101547 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.101555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.101559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.101717 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.102143 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.102148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.102151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.102663 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.102805 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.103365 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.103614 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:58:09.103993 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.082666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50566)
0620 06:58:09.087040 (+ 4374us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.087083 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.087286 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.087763 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.087771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.088631 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.089030 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.089048 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.090445 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.090451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.092026 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.092041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.093200 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.093207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.093415 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.094139 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.094161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.096951 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.100819 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.100831 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.100839 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.101402 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.101859 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.101864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.101868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.102017 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.102815 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.102823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.103020 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.103226 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.103553 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4121,"thread_start_us":3981,"threads_started":1}
I20260620 06:58:09.106340 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:32775: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9feaf91cbec040869784ceec0cbab59f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 32775
}
attrs {
promote: true
}
}
dest_uuid: "9cfb2df0979b4700aabe071cf98f4838"
cas_config_opid_index: -1
I20260620 06:58:09.107347 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } }
I20260620 06:58:09.111838 7114 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9feaf91cbec040869784ceec0cbab59f (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } }
I20260620 06:58:09.113005 7059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:09.115962 7114 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:32775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 06:58:09.116367 7114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } }
I20260620 06:58:09.116571 7114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:09.116667 7017 master.cc:537] Master@127.0.97.61:32775 shutting down...
I20260620 06:58:09.118276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44669
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:58:09.124013 7059 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Refusing update from remote peer 9cfb2df0979b4700aabe071cf98f4838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 06:58:09.125905 7114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:09.128794 7017 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 06:58:09.128880 6543 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 06:58:09.129165 7017 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:09.129331 7017 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: stopping tablet replica
W20260620 06:58:09.130664 6543 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 -> Peer 9feaf91cbec040869784ceec0cbab59f (127.0.97.61:32775): Couldn't send request to peer 9feaf91cbec040869784ceec0cbab59f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:58:09.136049 7017 master.cc:552] Master@127.0.97.61:32775 shutdown complete.
I20260620 06:58:09.136133 7017 master_runner.cc:305] Clearing existing system tablet
I20260620 06:58:09.136472 7017 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:09.136555 7017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:09.143124 7017 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.006s
I20260620 06:58:09.144446 7119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:09.144735 7017 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:09.144989 7017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:58:09.145064 7017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "9feaf91cbec040869784ceec0cbab59f"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "4bf7761b98eb73e4cb7c3d0bf050410f"
server_key_iv: "41acfabfe730af6cb5ec58b16247c638"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.148710 7017 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:09.153086 7017 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:58:09.153227 7017 log.cc:1196] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 06:58:09.153597 7017 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Deleting consensus metadata
I20260620 06:58:09.153833 7017 master_runner.cc:315] Copying system tablet from 127.0.97.62:37021
I20260620 06:58:09.162926 7017 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37021
I20260620 06:58:09.177258 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.163676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.0.97.62:37021)
0620 06:58:09.163909 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.163916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.167043 (+ 3127us) server_negotiation.cc:407] Connection header received
0620 06:58:09.167247 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.167252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.167330 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.167413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.168217 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.168981 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.170104 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.170517 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.170658 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.172405 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.172410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.172413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.172444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.174868 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.175459 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.175465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.175467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.175576 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.175916 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.175921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.175924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.176192 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.176325 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.176687 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.176914 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:58:09.177343 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.163573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39194)
0620 06:58:09.166933 (+ 3360us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.166949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.167094 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.167476 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.167480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.167868 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.168081 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.168090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.169111 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.169117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.169977 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.169988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.170668 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.170674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.170857 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.171420 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.171443 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.172220 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.174982 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.174987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.174991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.175358 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.175677 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.175683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.175687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.175820 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.176301 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.176306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.176466 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.176591 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.176924 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3234,"thread_start_us":3138,"threads_started":1}
I20260620 06:58:09.178565 6576 tablet_copy_service.cc:140] P 9cfb2df0979b4700aabe071cf98f4838: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9feaf91cbec040869784ceec0cbab59f ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39194)
I20260620 06:58:09.178838 6576 tablet_copy_service.cc:161] P 9cfb2df0979b4700aabe071cf98f4838: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9feaf91cbec040869784ceec0cbab59f at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39194: session id = 9feaf91cbec040869784ceec0cbab59f-00000000000000000000000000000000
I20260620 06:58:09.181669 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:58:09.183512 7017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:58:09.187953 7017 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: tablet copy: Starting download of 0 data blocks...
I20260620 06:58:09.188211 7017 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: tablet copy: Starting download of 1 WAL segments...
I20260620 06:58:09.190510 7017 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:58:09.193723 6576 tablet_copy_service.cc:342] P 9cfb2df0979b4700aabe071cf98f4838: Request end of tablet copy session 9feaf91cbec040869784ceec0cbab59f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:39194
I20260620 06:58:09.193888 6576 tablet_copy_service.cc:433] P 9cfb2df0979b4700aabe071cf98f4838: ending tablet copy session 9feaf91cbec040869784ceec0cbab59f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9feaf91cbec040869784ceec0cbab59f
I20260620 06:58:09.202935 7017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:09.205201 7126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.211387 7127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.215626 7129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:09.216022 7017 server_base.cc:1034] running on GCE node
I20260620 06:58:09.216315 7017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:09.216836 7017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:09.218123 7017 hybrid_clock.cc:648] HybridClock initialized: now 1781938689218060 us; error 35 us; skew 500 ppm
I20260620 06:58:09.218405 7017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:09.220270 7017 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 06:58:09.221434 7017 webserver.cc:460] Webserver started at http://127.0.97.61:35491/ using document root <none> and password file <none>
I20260620 06:58:09.221712 7017 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:09.221782 7017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:09.224090 7017 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:58:09.225570 7136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:09.225932 7017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:58:09.226054 7017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "9feaf91cbec040869784ceec0cbab59f"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "4bf7761b98eb73e4cb7c3d0bf050410f"
server_key_iv: "41acfabfe730af6cb5ec58b16247c638"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.226187 7017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:09.238186 7017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:09.241403 7017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:09.259290 7017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:32775
I20260620 06:58:09.259290 7187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:32775 every 8 connection(s)
I20260620 06:58:09.260175 7017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
I20260620 06:58:09.265517 7188 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:58:09.269515 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Bootstrap starting.
I20260620 06:58:09.283380 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:58:09.283795 7188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: Bootstrap complete.
I20260620 06:58:09.284603 7188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } }
I20260620 06:58:09.284797 7188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9feaf91cbec040869784ceec0cbab59f, State: Initialized, Role: LEARNER
I20260620 06:58:09.285003 7188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } }
I20260620 06:58:09.285701 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } }
I20260620 06:58:09.285955 7188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:09.286167 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: LEARNER
I20260620 06:58:09.300449 7202 catalog_manager.cc:1259] Loaded cluster ID: afdf28a77a0a41679c5939805be3bde4
I20260620 06:58:09.300560 7202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: loading cluster ID for follower catalog manager: success
I20260620 06:58:09.303097 7202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: acquiring CA information for follower catalog manager: success
I20260620 06:58:09.304730 7202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:58:09.510829 7012 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
W20260620 06:58:09.563328 7116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:09.563700 7116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:09.563841 7116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:09.575134 7116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:09.575284 7116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:09.575346 7116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:09.575433 7116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:09.575491 7116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:09.587284 7116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:09.588532 7116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:09.589737 7116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:09.602195 7210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.602264 7207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.602283 7208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:09.603099 7116 server_base.cc:1034] running on GCE node
I20260620 06:58:09.603837 7116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:09.605285 7116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:09.606473 7116 hybrid_clock.cc:648] HybridClock initialized: now 1781938689606418 us; error 47 us; skew 500 ppm
I20260620 06:58:09.606801 7116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:09.609676 7116 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 06:58:09.611107 7116 webserver.cc:460] Webserver started at http://127.0.97.60:36541/ using document root <none> and password file <none>
I20260620 06:58:09.611737 7116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:09.611846 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:09.612123 7116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:09.614972 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/instance:
uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
format_stamp: "Formatted at 2026-06-20 06:58:09 on dist-test-slave-mz20"
server_key: "051f906df01c4a1a6b1edec6e9699fdd"
server_key_iv: "9956b7e53663dafb501ac42f54224350"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.615680 7116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal/instance:
uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
format_stamp: "Formatted at 2026-06-20 06:58:09 on dist-test-slave-mz20"
server_key: "051f906df01c4a1a6b1edec6e9699fdd"
server_key_iv: "9956b7e53663dafb501ac42f54224350"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.620960 7116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:58:09.624398 7217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:09.625661 7116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:09.625804 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
format_stamp: "Formatted at 2026-06-20 06:58:09 on dist-test-slave-mz20"
server_key: "051f906df01c4a1a6b1edec6e9699fdd"
server_key_iv: "9956b7e53663dafb501ac42f54224350"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.625980 7116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:09.638901 7116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:09.642623 7116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:09.642863 7116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:09.660312 7116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44669
I20260620 06:58:09.660377 7268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44669 every 8 connection(s)
I20260620 06:58:09.661645 7116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
I20260620 06:58:09.665503 7269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:09.667004 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7116
I20260620 06:58:09.675004 7269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:09.690529 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.669126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.0.97.60:44669)
0620 06:58:09.669869 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.669881 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.669921 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:58:09.671884 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.671906 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.672314 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.672643 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.674491 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.675822 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.677462 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.678362 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.678560 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.683416 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.683426 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.683437 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.683477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.686401 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.687185 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.687194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.687207 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.687388 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.687706 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.687711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.687715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.688189 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.688507 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.688856 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.689259 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":341,"threads_started":1}
I20260620 06:58:09.693054 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.677916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.0.97.62:37021)
0620 06:58:09.678121 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.678128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.678769 (+ 641us) server_negotiation.cc:407] Connection header received
0620 06:58:09.679049 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.679054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.679130 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.679226 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.680345 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.681404 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.683706 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.684253 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.684371 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.687735 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.687739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.687742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.687771 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.690182 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.691049 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.691053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.691056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.691197 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.691506 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.691511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.691514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.691822 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.691951 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.692587 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.692791 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:58:09.693279 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.677804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39204)
0620 06:58:09.678626 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.678688 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.678903 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.679298 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.679305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.679975 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.680200 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.680208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.682021 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.682026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.683564 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.683576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.684425 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.684429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.684562 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.685133 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.685156 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.687592 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.690306 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.690315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.690322 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.690928 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.691300 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.691304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.691306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.691414 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.692025 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.692033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.692213 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.692507 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.692818 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":376,"threads_started":1}
I20260620 06:58:09.694682 7269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 06:58:09.707577 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.695326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50568)
0620 06:58:09.695495 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.695512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.695612 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.696101 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.696104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.696915 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.697042 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.697050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.698618 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.698624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.699592 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.699603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.700256 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.700261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.700438 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.700938 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.700954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.702787 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.705426 (+ 2639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.705431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.705435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.705772 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.706119 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.706124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.706128 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.706250 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.706724 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.706728 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.706862 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.706973 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.707248 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:58:09.707605 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.695415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.0.97.61:32775)
0620 06:58:09.695858 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.695862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.695876 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:58:09.695920 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.695923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.695976 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.696057 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.697170 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.698470 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.699719 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.700117 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.700219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.702960 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.702965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.702968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.703002 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.705312 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.705872 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.705877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.705879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.706023 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.706341 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.706345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.706347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.706625 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.706757 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.707065 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.707252 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":245,"threads_started":1}
I20260620 06:58:09.708804 7269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 06:58:09.720264 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.709561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.0.1:56692)
0620 06:58:09.709727 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.709742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.709820 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.710042 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.710044 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.710349 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.710483 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.710491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.711409 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.711412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.712218 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.712227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.712844 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.712848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.713094 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.713536 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.713550 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.715405 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.717829 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.717833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.717836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.718646 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.718986 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.718989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.718991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.719121 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.719536 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.719539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.719676 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.719777 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.719970 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:58:09.720316 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.709551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.0.97.60:44669)
0620 06:58:09.709705 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.709711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.709781 (+ 70us) server_negotiation.cc:407] Connection header received
0620 06:58:09.709903 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.709906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.709957 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.710022 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.710622 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.711307 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.712317 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.712715 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.712816 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.715550 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.715555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.715558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.715583 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.717686 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.718731 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.718734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.718736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.718855 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.719207 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.719210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.719212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.719453 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.719586 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.719841 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.720067 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":44}
I20260620 06:58:09.724725 7269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Bootstrap starting.
I20260620 06:58:09.727998 7269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:09.728971 7269 log.cc:824] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:09.731586 7269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: No bootstrap required, opened a new log
I20260620 06:58:09.739071 7269 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } }
I20260620 06:58:09.739374 7269 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:09.739475 7269 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc0fb9a2e8c4e1d904fff3fdc98bfd8, State: Initialized, Role: FOLLOWER
I20260620 06:58:09.740123 7269 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } }
I20260620 06:58:09.740371 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.734931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.97.61:32775)
0620 06:58:09.735111 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.735117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.735312 (+ 195us) server_negotiation.cc:407] Connection header received
0620 06:58:09.735502 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.735506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.735573 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.735659 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:09.736485 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.737165 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.738638 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.739426 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.739547 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.739711 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.740027 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.740135 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:58:09.740494 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.734659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50584)
0620 06:58:09.735223 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.735238 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.735378 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.735700 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.735705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.735730 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:09.736345 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.736355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.737291 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.737296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.738489 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.738502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.739631 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.739637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.739822 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.739883 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.739982 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.740148 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
I20260620 06:58:09.741245 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } } }
I20260620 06:58:09.741539 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:09.741489 7163 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 06:58:09.742038 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfb2df0979b4700aabe071cf98f4838. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } }
I20260620 06:58:09.742264 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: LEARNER
I20260620 06:58:09.742524 7269 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:58:09.751019 7289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:58:09.751138 7289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:58:09.754015 7116 master_runner.cc:428] Detected that this master cbc0fb9a2e8c4e1d904fff3fdc98bfd8 is joining an existing cluster
I20260620 06:58:09.754518 7116 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:44669
I20260620 06:58:09.771152 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.758409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.0.97.62:37021)
0620 06:58:09.758577 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.758582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.759199 (+ 617us) server_negotiation.cc:407] Connection header received
0620 06:58:09.759351 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.759356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.759431 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.759508 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.760450 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.761172 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.762982 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.763593 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.763699 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.765437 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.765442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.765445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.765475 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.768152 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.768898 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.768904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.768907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.769050 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.769512 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.769517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.769520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.769878 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.769985 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.770596 (+ 611us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.770809 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 06:58:09.772323 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.757980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39208)
0620 06:58:09.759101 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.759119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.759244 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.759545 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.759549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.760049 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.760313 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.760322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.761305 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.761310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.762854 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.762866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.763769 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.763776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.763937 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.764481 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.764500 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.765240 (+ 740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.768289 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.768297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.768301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.768774 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.769202 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.769207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.769211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.769397 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.770067 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.770071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.770295 (+ 224us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.771568 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.771860 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":843,"threads_started":1}
I20260620 06:58:09.782325 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.758144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.0.1:56704)
0620 06:58:09.767003 (+ 8859us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.767025 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.767158 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.767593 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.767597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.768243 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.768581 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.768596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.770005 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.770010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.771487 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.771500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.772619 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.772624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.772831 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.773405 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.773426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.774428 (+ 1002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.779394 (+ 4966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.779399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.779403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.779996 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.780398 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.780402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.780405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.780563 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.781254 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.781259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.781413 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.781554 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.781832 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8699,"mutex_wait_us":24,"thread_start_us":8526,"threads_started":1}
I20260620 06:58:09.782927 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.757949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.0.97.61:32775)
0620 06:58:09.758105 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.758111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.767691 (+ 9580us) server_negotiation.cc:407] Connection header received
0620 06:58:09.767923 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.767928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.768015 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.768081 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.769434 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.770965 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.772959 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.773534 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.773624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.776001 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.776008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.776012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.776050 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.779085 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.780506 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.780512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.780516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.780684 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.781127 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.781132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.781136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.781533 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.781649 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.782281 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.782563 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:58:09.783281 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.758241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.97.60:44669)
0620 06:58:09.758391 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.758396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.767120 (+ 8724us) server_negotiation.cc:407] Connection header received
0620 06:58:09.767257 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.767263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.767339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.767443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.768735 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.769788 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.771612 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.772253 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.772361 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.774638 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.774645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.774649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.774684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.779160 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.780128 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.780133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.780137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.780289 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.780668 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.780672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.780676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.781048 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.781204 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.782738 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.782943 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 06:58:09.784852 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.758326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50588)
0620 06:58:09.767602 (+ 9276us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.767617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.767780 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.768267 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.768271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.768938 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.769268 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.769280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.771132 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.771138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.772787 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.772801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.773912 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.773919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.774101 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.774707 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.774730 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.775802 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.779297 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.779302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.779307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.779972 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.780837 (+ 865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.780842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.780846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.781009 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.781748 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.781753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.781980 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.783582 (+ 1602us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.783916 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9167,"mutex_wait_us":32,"thread_start_us":9040,"threads_started":1}
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 06:58:09.796622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6617
I20260620 06:58:09.799607 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.778050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.0.1:56710)
0620 06:58:09.778374 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.778391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.778485 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.785162 (+ 6677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.785168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.785630 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.785815 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.785828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.787572 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.787576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.788445 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.788458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.789836 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.789840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.789977 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.790608 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.790625 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.792554 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.796802 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.796808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.796812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.797348 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.797755 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.797760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.797764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.797920 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.798569 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.798575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.798769 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.798921 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.799220 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260620 06:58:09.799615 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.778986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.97.60:44669)
0620 06:58:09.783489 (+ 4503us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.783494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.783511 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:58:09.783595 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.783600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.783700 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.783792 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.786263 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.787438 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.788595 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.789661 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.789798 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.792777 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.792784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.792789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.792824 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.796655 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.797500 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.797509 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.797512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.797636 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.798074 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.798079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.798082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.798417 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.798567 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.799034 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.799257 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4411,"thread_start_us":7744,"threads_started":1}
I20260620 06:58:09.801376 6556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:44669: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 44669
}
attrs {
promote: true
}
}
dest_uuid: "9cfb2df0979b4700aabe071cf98f4838"
cas_config_opid_index: 5
I20260620 06:58:09.802366 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } }
I20260620 06:58:09.804067 7276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.810774 7276 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:44669
I20260620 06:58:09.812000 7116 master.cc:537] Master@127.0.97.60:44669 shutting down...
I20260620 06:58:09.812184 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Refusing update from remote peer 9cfb2df0979b4700aabe071cf98f4838: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:58:09.813331 7300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 06:58:09.821421 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.825767 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.826025 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: LEARNER
W20260620 06:58:09.829171 6542 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 -> Peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60:44669): Couldn't send request to peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:58:09.829355 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.797475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.97.1:41147)
0620 06:58:09.807164 (+ 9689us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.807170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.807195 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:58:09.807261 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.807267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.807491 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.808980 (+ 1489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.809156 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.810113 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.811844 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.812705 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.812824 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.820507 (+ 7683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.820526 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.820545 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.820601 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.825098 (+ 4497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.826265 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.826275 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.826279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.826422 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.826842 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.826847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.826850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.827328 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.827629 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.828658 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.828989 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9527,"thread_start_us":9382,"threads_started":1}
I20260620 06:58:09.830427 7276 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: attempting to promote NON_VOTER 9feaf91cbec040869784ceec0cbab59f to VOTER
I20260620 06:58:09.830533 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: true } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.830884 6740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:09.830953 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:09.831830 7276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } }
I20260620 06:58:09.834416 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 LEARNER]: Refusing update from remote peer 9cfb2df0979b4700aabe071cf98f4838: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260620 06:58:09.834982 6542 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 -> Peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60:44669): Couldn't send request to peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:58:09.835196 7276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 06:58:09.837854 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9feaf91cbec040869784ceec0cbab59f (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.838579 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9feaf91cbec040869784ceec0cbab59f (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.843873 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.844106 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:09.844641 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.844919 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:09.845182 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:09.845396 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:09.848502 7116 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:09.849028 7116 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:09.850358 7116 tablet_replica.cc:330] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: stopping tablet replica
I20260620 06:58:09.859776 7116 master.cc:552] Master@127.0.97.60:44669 shutdown complete.
I20260620 06:58:09.859861 7116 master_runner.cc:305] Clearing existing system tablet
I20260620 06:58:09.860237 7116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:09.860344 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:09.869051 7116 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.006s
I20260620 06:58:09.874868 7308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:09.875198 7116 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:09.875433 7116 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.005s
I20260620 06:58:09.875533 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
format_stamp: "Formatted at 2026-06-20 06:58:09 on dist-test-slave-mz20"
server_key: "051f906df01c4a1a6b1edec6e9699fdd"
server_key_iv: "9956b7e53663dafb501ac42f54224350"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.878927 7116 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:09.883356 7116 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:58:09.883490 7116 log.cc:1196] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 06:58:09.883860 7116 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Deleting consensus metadata
I20260620 06:58:09.884121 7116 master_runner.cc:315] Copying system tablet from 127.0.97.62:37021
I20260620 06:58:09.886926 7116 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:37021
I20260620 06:58:09.908217 6747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.887663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.0.97.62:37021)
0620 06:58:09.887903 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:09.887921 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:09.891094 (+ 3173us) server_negotiation.cc:407] Connection header received
0620 06:58:09.891272 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:09.891276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:09.891338 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:09.891423 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:09.892307 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.893288 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.894790 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.895363 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.895462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.903110 (+ 7648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:09.903117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:09.903122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:09.903162 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:09.905770 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.906372 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.906377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.906381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.906517 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:09.906890 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:09.906896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:09.906899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:09.907146 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:09.907293 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:09.907662 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 06:58:09.907861 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 06:58:09.908259 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:09.887646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39212)
0620 06:58:09.890980 (+ 3334us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:09.890999 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:09.891145 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:09.891463 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:09.891468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:09.891885 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:09.892169 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.892179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.893475 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.893482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.894622 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:09.894638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:09.895771 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:09.895778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:09.898116 (+ 2338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:09.901730 (+ 3614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:09.901754 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:09.902849 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:09.905897 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.905902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.905906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.906254 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.906637 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:09.906642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:09.906645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:09.906783 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:09.907251 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:09.907256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:09.907425 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 06:58:09.907535 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:58:09.907843 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3200,"thread_start_us":3085,"threads_started":1}
I20260620 06:58:09.909153 6576 tablet_copy_service.cc:140] P 9cfb2df0979b4700aabe071cf98f4838: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39212)
I20260620 06:58:09.909305 6576 tablet_copy_service.cc:161] P 9cfb2df0979b4700aabe071cf98f4838: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39212: session id = cbc0fb9a2e8c4e1d904fff3fdc98bfd8-00000000000000000000000000000000
I20260620 06:58:09.911561 6576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:58:09.913244 7116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:58:09.918314 7116 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: tablet copy: Starting download of 0 data blocks...
I20260620 06:58:09.918612 7116 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: tablet copy: Starting download of 1 WAL segments...
I20260620 06:58:09.920713 7116 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:58:09.923765 6576 tablet_copy_service.cc:342] P 9cfb2df0979b4700aabe071cf98f4838: Request end of tablet copy session cbc0fb9a2e8c4e1d904fff3fdc98bfd8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:39212
I20260620 06:58:09.923882 6576 tablet_copy_service.cc:433] P 9cfb2df0979b4700aabe071cf98f4838: ending tablet copy session cbc0fb9a2e8c4e1d904fff3fdc98bfd8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8
I20260620 06:58:09.933019 7116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:09.935348 7315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.940064 7316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:09.947021 7318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:09.947655 7116 server_base.cc:1034] running on GCE node
I20260620 06:58:09.948004 7116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:09.948611 7116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:09.949800 7116 hybrid_clock.cc:648] HybridClock initialized: now 1781938689949740 us; error 52 us; skew 500 ppm
I20260620 06:58:09.950093 7116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:09 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938689, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:09.951874 7116 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 06:58:09.959775 7116 webserver.cc:460] Webserver started at http://127.0.97.60:36153/ using document root <none> and password file <none>
I20260620 06:58:09.960060 7116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:09.960148 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:09.962513 7116 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:58:09.964450 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:09.964862 7116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:58:09.964987 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8"
format_stamp: "Formatted at 2026-06-20 06:58:09 on dist-test-slave-mz20"
server_key: "051f906df01c4a1a6b1edec6e9699fdd"
server_key_iv: "9956b7e53663dafb501ac42f54224350"
server_key_version: "encryptionkey@0"
I20260620 06:58:09.965152 7116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:09.978991 7116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:09.982726 7116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:09.999203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6617
I20260620 06:58:10.016242 7376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44669 every 8 connection(s)
I20260620 06:58:10.016455 7116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44669
I20260620 06:58:10.017215 7116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
I20260620 06:58:10.021608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41147
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37021
--webserver_port=34891
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:58:10.027151 7377 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:58:10.031396 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Bootstrap starting.
I20260620 06:58:10.051801 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:58:10.052234 7377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Bootstrap complete.
I20260620 06:58:10.053090 7377 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } }
I20260620 06:58:10.053306 7377 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbc0fb9a2e8c4e1d904fff3fdc98bfd8, State: Initialized, Role: LEARNER
I20260620 06:58:10.053539 7377 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } }
I20260620 06:58:10.054739 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:10.055119 7377 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:10.055126 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:58:10.083277 7392 catalog_manager.cc:1259] Loaded cluster ID: afdf28a77a0a41679c5939805be3bde4
I20260620 06:58:10.083384 7392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: loading cluster ID for follower catalog manager: success
I20260620 06:58:10.085877 7392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: acquiring CA information for follower catalog manager: success
I20260620 06:58:10.087862 7392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:58:10.276011 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.263242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.0.1:56716)
0620 06:58:10.263613 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:10.263632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:10.263780 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:10.271352 (+ 7572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:10.271358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:10.271393 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:10.271558 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.271568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.272862 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.272866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.274041 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.274056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.275240 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.275246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.275444 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.275477 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:58:10.275570 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:58:10.275711 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260620 06:58:10.276015 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.264073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.60:44669)
0620 06:58:10.270996 (+ 6923us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:10.271000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:10.271020 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:58:10.271090 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:10.271094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:10.271166 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:10.271287 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:10.271711 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.272688 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.274201 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.275074 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.275194 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.275425 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:10.275594 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 06:58:10.275717 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6807,"thread_start_us":6657,"threads_started":1}
I20260620 06:58:10.277052 7352 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 06:58:10.277673 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cfb2df0979b4700aabe071cf98f4838. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:10.277951 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: LEARNER
W20260620 06:58:10.410030 7379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:10.410315 7379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:10.410416 7379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:10.418174 7379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:10.418298 7379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:10.418418 7379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:10.428570 7379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34891
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:10.429843 7379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:10.431176 7379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:10.443640 7398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:10.443606 7401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:10.443665 7399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:10.444608 7379 server_base.cc:1034] running on GCE node
I20260620 06:58:10.445626 7379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:10.446492 7379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:10.447719 7379 hybrid_clock.cc:648] HybridClock initialized: now 1781938690447673 us; error 39 us; skew 500 ppm
I20260620 06:58:10.448022 7379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:10 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:10.450978 7379 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 06:58:10.452365 7379 webserver.cc:460] Webserver started at http://127.0.97.1:34891/ using document root <none> and password file <none>
I20260620 06:58:10.453006 7379 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:10.453130 7379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:10.458379 7379 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:58:10.461202 7408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:10.462698 7379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:10.462852 7379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "2e4d6d1c44f04db5ab57c8ca75c360de"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "2ca8fb86ecd929dc1fe036acb941e663"
server_key_iv: "e5abbce2fb85caac617838f89f19c15c"
server_key_version: "encryptionkey@0"
I20260620 06:58:10.463840 7379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:10.477715 7379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:10.481884 7379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:10.482126 7379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:10.483290 7379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:10.484891 7379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:10.485004 7379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:10.485116 7379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:10.485188 7379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:10.517354 7379 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41147
I20260620 06:58:10.517432 7520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41147 every 8 connection(s)
I20260620 06:58:10.518831 7379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:10.527482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7379
I20260620 06:58:10.527830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6751
Jun 20 06:58:10 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 06:58:10 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 06:58:10 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 06:58:10 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938690, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:10.570521 7393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.539764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55499 (local address 127.0.97.60:44669)
0620 06:58:10.539962 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:10.539969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:10.553668 (+ 13699us) server_negotiation.cc:407] Connection header received
0620 06:58:10.553780 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:10.553786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:10.553869 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:10.553972 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:10.555208 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.556238 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.558010 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.558643 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.558754 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.563045 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:10.563052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:10.563057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:10.563111 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:10.567107 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568020 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.568026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.568029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.568167 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568738 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.568743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.568746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.569064 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:10.569176 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:10.569964 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 06:58:10.570191 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:58:10.570653 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.522259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39005 (local address 127.0.97.61:32775)
0620 06:58:10.545939 (+ 23680us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:10.545947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:10.550031 (+ 4084us) server_negotiation.cc:407] Connection header received
0620 06:58:10.550523 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:10.550530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:10.550673 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:10.550757 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:10.553932 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.555037 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.557215 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.557852 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.557946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.563434 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:10.563442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:10.563446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:10.563490 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:10.566760 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.567771 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.567777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.567781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.567918 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568470 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.568475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.568478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.568816 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:10.568925 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:10.570071 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0620 06:58:10.570277 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23548,"thread_start_us":23701,"threads_started":1}
I20260620 06:58:10.571719 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.521973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47297 (local address 127.0.97.62:37021)
0620 06:58:10.546344 (+ 24371us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:10.546349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:10.547844 (+ 1495us) server_negotiation.cc:407] Connection header received
0620 06:58:10.550456 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:10.550462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:10.550584 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:10.550684 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:10.553444 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.554570 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.557224 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.557854 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.557955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.563817 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:10.563823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:10.563828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:10.563874 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:10.567992 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568853 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.568859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.568862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.569002 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.569817 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.569821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.569824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.570152 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:10.570258 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:10.571178 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 06:58:10.571386 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24267,"thread_start_us":24173,"threads_started":1}
I20260620 06:58:10.571851 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.529035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.0.97.2:46257)
0620 06:58:10.545358 (+ 16323us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:10.545369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:10.547177 (+ 1808us) server_negotiation.cc:407] Connection header received
0620 06:58:10.548759 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:10.548769 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:10.549031 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:10.549188 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:10.551449 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.552951 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.554870 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.555715 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.555812 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.564129 (+ 8317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:10.564151 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:10.564170 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:10.564218 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:10.568052 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568935 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.568945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.568948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.569115 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:10.569956 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:10.569961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:10.569964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:10.570349 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:10.570660 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:10.571230 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 06:58:10.571467 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16111,"thread_start_us":16347,"threads_started":1}
I20260620 06:58:10.572896 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.539083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.97.1:55499)
0620 06:58:10.553540 (+ 14457us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:10.553559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:10.553675 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:10.554142 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:10.554146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:10.554719 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:10.555032 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.555043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.556404 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.556409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.557869 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.557881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.558980 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.558986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.559181 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.559703 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:10.559722 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:10.562788 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:10.567261 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.567266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.567270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.567884 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.568322 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.568329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.568597 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.569244 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:10.569248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:10.569475 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 06:58:10.570839 (+ 1364us) client_negotiation.cc:240] Negotiation successful
0620 06:58:10.571585 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14335,"mutex_wait_us":214,"thread_start_us":8227,"threads_started":1}
I20260620 06:58:10.572928 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.522712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.97.1:39005)
0620 06:58:10.549907 (+ 27195us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:10.549934 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:10.550359 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:10.551168 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:10.551182 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:10.552137 (+ 955us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:10.553768 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.553777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.555243 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.555246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.557046 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.557060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.558297 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.558303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.558722 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.559510 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:10.559537 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:10.563145 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:10.566930 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.566940 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.566961 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.567622 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.568072 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.568079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.568337 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.569033 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:10.569042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:10.569796 (+ 754us) client_negotiation.cc:769] Sending connection context
0620 06:58:10.570892 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0620 06:58:10.571433 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27002,"mutex_wait_us":787,"spinlock_wait_cycles":27008,"thread_start_us":27183,"threads_started":1}
I20260620 06:58:10.573386 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:10.522712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:47297)
0620 06:58:10.547677 (+ 24965us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:10.547731 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:10.549990 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:10.551168 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:10.551182 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:10.552136 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:10.553224 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.553252 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.554761 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.554765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.557043 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:10.557056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:10.558297 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:10.558304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:10.558727 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:10.559559 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:10.559581 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:10.563547 (+ 3966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:10.568239 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.568244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.568247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.568730 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.569156 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:10.569160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:10.569164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:10.569691 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:10.570346 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:10.570351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:10.570486 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 06:58:10.572298 (+ 1812us) client_negotiation.cc:240] Negotiation successful
0620 06:58:10.572564 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24687,"mutex_wait_us":110,"spinlock_wait_cycles":9600,"thread_start_us":30445,"threads_started":1}
I20260620 06:58:10.573972 6874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:10.574613 7523 heartbeater.cc:347] Connected to a master server at 127.0.97.61:32775
I20260620 06:58:10.574898 7523 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:10.575734 7523 heartbeater.cc:511] Master 127.0.97.61:32775 requested a full tablet report, sending...
I20260620 06:58:10.576507 7521 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44669
I20260620 06:58:10.576620 7521 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:10.577000 7521 heartbeater.cc:511] Master 127.0.97.60:44669 requested a full tablet report, sending...
I20260620 06:58:10.577265 7522 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260620 06:58:10.577368 7522 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:10.577755 7522 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:10.578922 6556 ts_manager.cc:194] Re-registered known tserver with Master: 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:10.578922 7342 ts_manager.cc:194] Registered new tserver with Master: 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:10.579567 7153 ts_manager.cc:194] Registered new tserver with Master: 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:10.579934 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47297
I20260620 06:58:10.658497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6751
I20260620 06:58:10.671478 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:46257
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37021
--webserver_port=35419
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:58:10.679308 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: true } } }
I20260620 06:58:10.679651 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:10.681267 7534 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: attempting to promote NON_VOTER cbc0fb9a2e8c4e1d904fff3fdc98bfd8 to VOTER
I20260620 06:58:10.681785 7534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } }
I20260620 06:58:10.683511 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Refusing update from remote peer 9cfb2df0979b4700aabe071cf98f4838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 06:58:10.683584 7352 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 LEARNER]: Refusing update from remote peer 9cfb2df0979b4700aabe071cf98f4838: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 06:58:10.684147 7535 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 06:58:10.684571 7534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 06:58:10.686905 7534 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.688766 7352 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.690482 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.690763 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:10.690956 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.691187 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:10.692225 7163 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.692453 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.692672 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:10.695168 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.695374 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:10.704356 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9cfb2df0979b4700aabe071cf98f4838" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:10.704741 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:58:10.988337 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:10.988612 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:10.988706 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:10.996155 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:10.996264 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:10.996412 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:11.005955 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:46257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=35419
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:11.007177 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:11.008380 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:11.021200 7545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:11.021147 7547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:11.021767 7544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:11.023044 7533 server_base.cc:1034] running on GCE node
I20260620 06:58:11.023671 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:11.024642 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:11.025871 7533 hybrid_clock.cc:648] HybridClock initialized: now 1781938691025799 us; error 55 us; skew 500 ppm
I20260620 06:58:11.026232 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:11.029372 7533 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 06:58:11.030812 7533 webserver.cc:460] Webserver started at http://127.0.97.2:35419/ using document root <none> and password file <none>
I20260620 06:58:11.031468 7533 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:11.031597 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:11.036866 7533 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 06:58:11.039724 7554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.041030 7533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:58:11.041220 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "d389b7db88f141fe9b406b6197844bdc"
format_stamp: "Formatted at 2026-06-20 06:58:07 on dist-test-slave-mz20"
server_key: "116f66366dea78ac9c9995f7741438e5"
server_key_iv: "e1019b651daf345baae18631c7f55e0b"
server_key_version: "encryptionkey@0"
I20260620 06:58:11.042148 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:11.056556 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:11.060259 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:11.060495 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:11.061390 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:11.062724 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:11.062831 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.062929 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:11.062999 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.092159 7533 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46257
I20260620 06:58:11.092203 7666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46257 every 8 connection(s)
I20260620 06:58:11.093662 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:11.102874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7533
I20260620 06:58:11.103132 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6884
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 06:58:11.158036 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.096570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55077 (local address 127.0.97.61:32775)
0620 06:58:11.109268 (+ 12698us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.109274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.138897 (+ 29623us) server_negotiation.cc:407] Connection header received
0620 06:58:11.139056 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.139062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.139145 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.139253 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.141925 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.142953 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145078 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.145697 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145787 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.150980 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.150986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.150991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.151039 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.154249 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155405 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.155410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.155415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.155566 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.156072 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.156076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.156079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.156380 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.156484 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.157472 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.157697 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12557,"thread_start_us":13980,"threads_started":1}
I20260620 06:58:11.158041 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.096628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38633 (local address 127.0.97.62:37021)
0620 06:58:11.113657 (+ 17029us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.113663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.139515 (+ 25852us) server_negotiation.cc:407] Connection header received
0620 06:58:11.139626 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.139631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.139705 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.139795 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.141567 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.142695 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145077 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.145702 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145800 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.150743 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.150750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.150754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.150793 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.154340 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155405 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.155410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.155414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.155566 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.156581 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.156586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.156589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.156883 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.157001 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.157491 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.157686 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16913,"thread_start_us":16807,"threads_started":1}
I20260620 06:58:11.158870 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.104400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.97.3:35001)
0620 06:58:11.134897 (+ 30497us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.134908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.136731 (+ 1823us) server_negotiation.cc:407] Connection header received
0620 06:58:11.136842 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.136852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.137144 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.137316 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.139129 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.140606 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.142391 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.143245 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.143362 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.151205 (+ 7843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.151221 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.151238 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.151279 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.154897 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155665 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.155674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.155677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.155815 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.156412 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.156417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.156419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.156806 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.157078 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.158130 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.158482 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30274,"thread_start_us":30519,"threads_started":1}
I20260620 06:58:11.159464 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.096806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53799 (local address 127.0.97.60:44669)
0620 06:58:11.137890 (+ 41084us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.137897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.137916 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:58:11.137972 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.137977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.138048 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.138156 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.141344 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.142412 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145080 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.145699 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.145818 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.151386 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.151393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.151397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.151435 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.154723 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155588 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.155593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.155597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.155734 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.156701 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.156706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.156709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.157012 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.157149 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.158715 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.159148 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40924,"thread_start_us":42769,"threads_started":1}
I20260620 06:58:11.159912 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.097605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:38633)
0620 06:58:11.139400 (+ 41795us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.139418 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.139521 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.139838 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.139843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.141088 (+ 1245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.141412 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.141421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.142851 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.142857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.144924 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.144938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.146175 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.146181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.146502 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.147302 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.147335 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.150466 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.154459 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.154463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.154466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.155241 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.155704 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.155712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.155961 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.157071 (+ 1110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.157075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.157224 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.158216 (+ 992us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.158471 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41661,"mutex_wait_us":616,"spinlock_wait_cycles":72576,"thread_start_us":41134,"threads_started":1}
I20260620 06:58:11.159929 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.097602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.97.2:55077)
0620 06:58:11.138774 (+ 41172us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.138801 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.138937 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.140048 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.140053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.141193 (+ 1140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.141738 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.141748 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.143108 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.143113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.144924 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.144938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.146141 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.146148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.146514 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.147362 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.147380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.150773 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.154410 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.154418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.154437 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.155241 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.155704 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.155712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.155961 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.156589 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.156597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.157005 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.158343 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.158588 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41002,"mutex_wait_us":418,"spinlock_wait_cycles":14976,"thread_start_us":40651,"threads_started":1}
I20260620 06:58:11.160231 7007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:11.160795 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.097602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.97.2:53799)
0620 06:58:11.109859 (+ 12257us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.109916 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.112757 (+ 2841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.138644 (+ 25887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.138658 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.140058 (+ 1400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.141139 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.141179 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.142631 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.142636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.144911 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.144925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.146126 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.146133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.146523 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.147439 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.147462 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.151183 (+ 3721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.154902 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.154907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.154910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.155441 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.155857 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.155865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.155868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.156577 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.157205 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.157209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.157378 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.157510 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.158025 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11937,"mutex_wait_us":235,"spinlock_wait_cycles":896,"thread_start_us":11706,"threads_started":1}
I20260620 06:58:11.161394 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.61:32775
I20260620 06:58:11.161515 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
I20260620 06:58:11.161710 7669 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.161715 7668 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.162111 7667 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44669
I20260620 06:58:11.162245 7667 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.162711 7668 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:11.162802 7667 heartbeater.cc:511] Master 127.0.97.60:44669 requested a full tablet report, sending...
I20260620 06:58:11.163590 7669 heartbeater.cc:511] Master 127.0.97.61:32775 requested a full tablet report, sending...
I20260620 06:58:11.164445 6556 ts_manager.cc:194] Re-registered known tserver with Master: d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257)
I20260620 06:58:11.164456 7342 ts_manager.cc:194] Registered new tserver with Master: d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257)
I20260620 06:58:11.164551 7153 ts_manager.cc:194] Registered new tserver with Master: d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257)
I20260620 06:58:11.165477 6556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38633
I20260620 06:58:11.249543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6884
I20260620 06:58:11.263919 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35001
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37021
--webserver_port=44065
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:58:11.582257 7522 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
W20260620 06:58:11.582116 7680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:11.582568 7680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:11.582751 7680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:11.590318 7680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:11.590431 7680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:11.590618 7680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:11.600744 7680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=44065
--tserver_master_addrs=127.0.97.62:37021,127.0.97.61:32775,127.0.97.60:44669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:11.602061 7680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:11.603346 7680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:11.616096 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:11.616144 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:11.616086 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:11.617715 7680 server_base.cc:1034] running on GCE node
I20260620 06:58:11.618331 7680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:11.619346 7680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:11.620573 7680 hybrid_clock.cc:648] HybridClock initialized: now 1781938691620496 us; error 67 us; skew 500 ppm
I20260620 06:58:11.620913 7680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:11.624094 7680 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 06:58:11.625545 7680 webserver.cc:460] Webserver started at http://127.0.97.3:44065/ using document root <none> and password file <none>
I20260620 06:58:11.626266 7680 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:11.626406 7680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:11.631978 7680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 06:58:11.634949 7695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.636426 7680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:58:11.636602 7680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "bd85221f3ac548a29967a47c0a955ee5"
format_stamp: "Formatted at 2026-06-20 06:58:08 on dist-test-slave-mz20"
server_key: "b5f49f9029be75e58d02e218ba5bdf63"
server_key_iv: "643b5dfdeaa264b92863362c814ba8a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:11.637598 7680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:11.648674 7680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:11.652819 7680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:11.653066 7680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:11.654003 7680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:11.655402 7680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:11.655493 7680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.655592 7680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:11.655658 7680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:11.686368 7680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35001
I20260620 06:58:11.686440 7807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35001 every 8 connection(s)
I20260620 06:58:11.687750 7680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:11.692836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7680
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 06:58:11.750334 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.703179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.0.97.60:44669)
0620 06:58:11.732912 (+ 29733us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.732919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.732937 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:58:11.732986 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.732991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.733061 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.733155 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.735652 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.736891 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.738758 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.739400 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.739514 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.743583 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.743590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.743594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.743636 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.746985 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.748130 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.748135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.748138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.748268 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.748631 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.748636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.748639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.748923 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.749136 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.749791 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.749986 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29615,"thread_start_us":36441,"threads_started":1}
I20260620 06:58:11.750326 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.701333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.97.62:37021)
0620 06:58:11.724349 (+ 23016us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.724355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.726814 (+ 2459us) server_negotiation.cc:407] Connection header received
0620 06:58:11.727724 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.727730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.727809 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.727882 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.732321 (+ 4439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.733404 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.737307 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.737941 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.738037 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.742995 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.743002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.743007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.743049 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.746699 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.747990 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.747996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.747999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.748153 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.748566 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.748571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.748574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.748907 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.749118 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.749745 (+ 627us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.749965 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22909,"thread_start_us":21385,"threads_started":1}
Jun 20 06:58:11 dist-test-slave-mz20 krb5kdc[6511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:11.754457 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.690756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43253 (local address 127.0.97.61:32775)
0620 06:58:11.724670 (+ 33914us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.724677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.724697 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:58:11.724759 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.724765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.724843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.724951 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.732074 (+ 7123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.733289 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.739058 (+ 5769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.740063 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.740186 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.745356 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.745363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.745367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.745408 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.748957 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.749952 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.749957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.749961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.750099 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.750684 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.750688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.750692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.751002 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.751129 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.753685 (+ 2556us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.754090 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33796,"thread_start_us":33705,"threads_started":1}
I20260620 06:58:11.756182 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.691095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.97.3:43253)
0620 06:58:11.723816 (+ 32721us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.723841 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.723947 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.725478 (+ 1531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.725494 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.726593 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.731889 (+ 5296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.731900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.737486 (+ 5586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.737491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.738900 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.738911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.740896 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.740907 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.741217 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.742027 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.742067 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.745107 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.749143 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.749152 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.749171 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.749798 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.750238 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.750243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.750246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.750489 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.751201 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.751210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.752357 (+ 1147us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.752707 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.754706 (+ 1999us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32549,"mutex_wait_us":389,"thread_start_us":26956,"threads_started":1}
I20260620 06:58:11.759964 7810 heartbeater.cc:347] Connected to a master server at 127.0.97.61:32775
I20260620 06:58:11.760288 7810 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.761162 7810 heartbeater.cc:511] Master 127.0.97.61:32775 requested a full tablet report, sending...
I20260620 06:58:11.762984 7153 ts_manager.cc:194] Registered new tserver with Master: bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:11.763725 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.691192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.97.3:43601)
0620 06:58:11.719807 (+ 28615us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.719828 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.719947 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.727553 (+ 7606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.727558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.728164 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.729131 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.729168 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.734942 (+ 5774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.734948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.737308 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.737326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.743507 (+ 6181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.743513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.743697 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.745814 (+ 2117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.745835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.753413 (+ 7578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.758888 (+ 5475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.758893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.758897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.759370 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.759791 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.759796 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.759800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.762001 (+ 2201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.762763 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.762768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.762905 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.763041 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.763321 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28485,"spinlock_wait_cycles":63744,"thread_start_us":28771,"threads_started":1}
I20260620 06:58:11.764566 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.690427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43601 (local address 127.0.97.60:44669)
0620 06:58:11.727053 (+ 36626us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.727059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.727081 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:58:11.727156 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.727162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.727242 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.727354 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.733490 (+ 6136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.734725 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.742446 (+ 7721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.743113 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.743246 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.753915 (+ 10669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.753921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.755117 (+ 1196us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.755157 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.758737 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.759505 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.759511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.759515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.759666 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.762138 (+ 2472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.762143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.762147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.762559 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.762692 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.763980 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.764202 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36484,"thread_start_us":41421,"threads_started":1}
I20260620 06:58:11.765369 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.60:44669
I20260620 06:58:11.765489 7808 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.766453 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.725907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.0.97.61:32775)
0620 06:58:11.738530 (+ 12623us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.738556 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.744017 (+ 5461us) server_negotiation.cc:407] Connection header received
0620 06:58:11.744075 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.744079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.744175 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.744272 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.752815 (+ 8543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.753920 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.756069 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.756734 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.756842 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.760410 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.760416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.760421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.760452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.763185 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.763976 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.763981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.763984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.764145 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.764579 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.764583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.764586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.764887 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.765127 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.765672 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.765930 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12499,"thread_start_us":12656,"threads_started":1}
I20260620 06:58:11.766475 7808 heartbeater.cc:511] Master 127.0.97.60:44669 requested a full tablet report, sending...
I20260620 06:58:11.767655 7342 ts_manager.cc:194] Registered new tserver with Master: bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:11.755913 6556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:58:11.769073 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.691095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:53185)
0620 06:58:11.715521 (+ 24426us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.715599 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.717997 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.729361 (+ 11364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.729366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.729799 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:11.730050 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.730060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.739063 (+ 9003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.739067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.740962 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.740973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.747751 (+ 6778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.747757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.747936 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.749740 (+ 1804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:11.749762 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:11.754124 (+ 4362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:11.766378 (+ 12254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.766384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.766388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.766958 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.767359 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:11.767363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:11.767367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:11.767521 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:11.768177 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:11.768182 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:11.768286 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.768384 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.768676 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24042,"thread_start_us":32295,"threads_started":1}
I20260620 06:58:11.769122 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.691630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53185 (local address 127.0.97.62:37021)
0620 06:58:11.722940 (+ 31310us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.722947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.722970 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:58:11.723035 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.723041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.723124 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.723243 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:11.730246 (+ 7003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.731465 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.746636 (+ 15171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.747377 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.747472 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.754340 (+ 6868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:11.754347 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:11.754351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:11.754387 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:11.765806 (+ 11419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.767088 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.767094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.767097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.767234 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:11.767661 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:11.767666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:11.767668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:11.767996 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:11.768114 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.768510 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.768715 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31182,"thread_start_us":32490,"threads_started":1}
I20260620 06:58:11.769941 7809 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37021
W20260620 06:58:11.769975 6556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:11.770056 7809 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:11.770434 7809 heartbeater.cc:511] Master 127.0.97.62:37021 requested a full tablet report, sending...
I20260620 06:58:11.771451 6555 ts_manager.cc:194] Re-registered known tserver with Master: bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:11.772583 6555 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53185
I20260620 06:58:11.822558 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.796422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47424)
0620 06:58:11.810435 (+ 14013us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.810453 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.810592 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.814097 (+ 3505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.814103 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.814132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:11.814520 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.814532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.817984 (+ 3452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.817989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.819604 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.819616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.821843 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.821849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.822007 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.822034 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.822082 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.822183 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13895,"thread_start_us":11023,"threads_started":1}
I20260620 06:58:11.822799 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.796356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.97.2:46257)
0620 06:58:11.814277 (+ 17921us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.814286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.814318 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:58:11.814417 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.814425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.814716 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.815732 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:11.815951 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.817344 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.819764 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.821249 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.821370 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.821563 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.822340 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.822501 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17730,"thread_start_us":18511,"threads_started":1}
I20260620 06:58:11.823110 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.796163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46257 (local address 127.0.0.1:57696)
0620 06:58:11.807767 (+ 11604us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.807791 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.807949 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.815122 (+ 7173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.815128 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.815170 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:11.815538 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.815549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.817476 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.817481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.819559 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.819571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.821709 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.821715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.821901 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.821935 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.822018 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.822162 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11433,"mutex_wait_us":4,"thread_start_us":21246,"threads_started":1}
I20260620 06:58:11.823961 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.796960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.97.1:41147)
0620 06:58:11.813111 (+ 16151us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.813123 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.813159 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:58:11.813292 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.813302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.813604 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.813843 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:11.815006 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.816648 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.819771 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.821650 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.823290 (+ 1640us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.823495 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.823555 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.823702 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15959,"thread_start_us":17746,"threads_started":1}
I20260620 06:58:11.824560 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.796870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35001 (local address 127.0.0.1:47868)
0620 06:58:11.817687 (+ 20817us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:11.817703 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:11.817827 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:11.818613 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:11.818618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:11.818646 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:11.819057 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.819068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.820776 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.820781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.822430 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:11.822440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.823867 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.823873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:11.824116 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.824149 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:58:11.824231 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:58:11.824348 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20691,"thread_start_us":14042,"threads_started":1}
I20260620 06:58:11.824725 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:11.797326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.0.97.3:35001)
0620 06:58:11.817244 (+ 19918us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:11.817253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:11.817808 (+ 555us) server_negotiation.cc:407] Connection header received
0620 06:58:11.817994 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:11.818002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:11.818245 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:11.819230 (+ 985us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:11.819485 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.820657 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.822584 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:11.823687 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:11.823796 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:11.823978 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:11.824285 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 06:58:11.824432 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19723,"thread_start_us":19555,"threads_started":1}
I20260620 06:58:11.827756 7743 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcecce26a2d4e23a9c578727574a367 (DEFAULT_TABLE table=test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:11.827746 7602 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcecce26a2d4e23a9c578727574a367 (DEFAULT_TABLE table=test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:11.829128 7602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcecce26a2d4e23a9c578727574a367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:11.829139 7743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcecce26a2d4e23a9c578727574a367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:11.837785 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 8dcecce26a2d4e23a9c578727574a367 (DEFAULT_TABLE table=test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:11.839228 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dcecce26a2d4e23a9c578727574a367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:11.847338 7845 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Bootstrap starting.
I20260620 06:58:11.848527 7846 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Bootstrap starting.
I20260620 06:58:11.852535 7846 tablet_bootstrap.cc:654] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:11.852569 7845 tablet_bootstrap.cc:654] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:11.853473 7847 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Bootstrap starting.
I20260620 06:58:11.853782 7846 log.cc:824] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:11.857546 7847 tablet_bootstrap.cc:654] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:11.858795 7847 log.cc:824] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:11.859357 7845 log.cc:824] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:11.866745 7846 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: No bootstrap required, opened a new log
I20260620 06:58:11.867121 7846 ts_tablet_manager.cc:1397] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20260620 06:58:11.870095 7847 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: No bootstrap required, opened a new log
I20260620 06:58:11.870366 7847 ts_tablet_manager.cc:1397] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260620 06:58:11.872110 7845 tablet_bootstrap.cc:492] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: No bootstrap required, opened a new log
I20260620 06:58:11.872357 7845 ts_tablet_manager.cc:1397] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.005s
I20260620 06:58:11.878207 7846 raft_consensus.cc:348] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.878702 7846 raft_consensus.cc:374] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:11.878798 7846 raft_consensus.cc:729] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389b7db88f141fe9b406b6197844bdc, State: Initialized, Role: FOLLOWER
I20260620 06:58:11.879418 7846 consensus_queue.cc:260] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.880003 7845 raft_consensus.cc:348] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.880394 7845 raft_consensus.cc:374] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:11.880483 7845 raft_consensus.cc:729] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd85221f3ac548a29967a47c0a955ee5, State: Initialized, Role: FOLLOWER
I20260620 06:58:11.880641 7668 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
I20260620 06:58:11.880615 7847 raft_consensus.cc:348] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.880995 7847 raft_consensus.cc:374] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:11.881074 7847 raft_consensus.cc:729] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d6d1c44f04db5ab57c8ca75c360de, State: Initialized, Role: FOLLOWER
I20260620 06:58:11.881093 7845 consensus_queue.cc:260] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.881659 7847 consensus_queue.cc:260] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:11.883052 7809 heartbeater.cc:503] Master 127.0.97.62:37021 was elected leader, sending a full tablet report...
I20260620 06:58:11.883250 7845 ts_tablet_manager.cc:1428] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 06:58:11.886868 7846 ts_tablet_manager.cc:1428] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Time spent starting tablet: real 0.020s user 0.009s sys 0.005s
I20260620 06:58:11.888218 7847 ts_tablet_manager.cc:1428] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
W20260620 06:58:11.946448 7811 tablet.cc:2369] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:58:12.022920 7524 tablet.cc:2369] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:58:12.076027 7852 raft_consensus.cc:482] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:12.076367 7852 raft_consensus.cc:504] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:12.077961 7852 leader_election.cc:288] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257), 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:12.085912 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.078349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52507 (local address 127.0.97.2:46257)
0620 06:58:12.078574 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.078581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.078673 (+ 92us) server_negotiation.cc:407] Connection header received
0620 06:58:12.078865 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.078869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.078946 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.079050 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:12.079457 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.080578 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.083587 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.084786 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.084918 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.085021 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.085551 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.085673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:58:12.085959 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.078248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46257 (local address 127.0.97.3:52507)
0620 06:58:12.078579 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:12.078599 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:12.078740 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:12.079074 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:12.079079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:12.079158 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:12.079320 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:12.079331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.080734 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.080739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:12.083437 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:12.083447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.084964 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.084969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:12.085294 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.085366 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 06:58:12.085461 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:58:12.085630 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":176,"spinlock_wait_cycles":10624}
I20260620 06:58:12.085999 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.078336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.97.3:49703)
0620 06:58:12.078605 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:12.078628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:12.078739 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:12.079074 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:12.079079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:12.079113 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:12.079333 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:12.079341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.080719 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.080724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:12.083430 (+ 2706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:12.083447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.084961 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.084969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:12.085294 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.085366 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 06:58:12.085461 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:58:12.085613 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":171,"spinlock_wait_cycles":20096}
I20260620 06:58:12.085950 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.078441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49703 (local address 127.0.97.1:41147)
0620 06:58:12.078638 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.078642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.078663 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:58:12.078864 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.078869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.078946 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.079050 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:12.079457 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.080578 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.083590 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.084786 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.084918 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.085024 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.085509 (+ 485us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.085638 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:58:12.087069 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcecce26a2d4e23a9c578727574a367" candidate_uuid: "bd85221f3ac548a29967a47c0a955ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389b7db88f141fe9b406b6197844bdc" is_pre_election: true
I20260620 06:58:12.087487 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcecce26a2d4e23a9c578727574a367" candidate_uuid: "bd85221f3ac548a29967a47c0a955ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" is_pre_election: true
I20260620 06:58:12.087664 7622 raft_consensus.cc:2454] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd85221f3ac548a29967a47c0a955ee5 in term 0.
I20260620 06:58:12.087845 7476 raft_consensus.cc:2454] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd85221f3ac548a29967a47c0a955ee5 in term 0.
I20260620 06:58:12.088359 7696 leader_election.cc:302] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd85221f3ac548a29967a47c0a955ee5, d389b7db88f141fe9b406b6197844bdc; no voters:
I20260620 06:58:12.089329 7852 raft_consensus.cc:2781] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:12.089448 7852 raft_consensus.cc:482] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:12.089540 7852 raft_consensus.cc:3037] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:12.092525 7852 raft_consensus.cc:504] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:12.093427 7852 leader_election.cc:288] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [CANDIDATE]: Term 1 election: Requested vote from peers d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257), 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147)
I20260620 06:58:12.095202 7476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcecce26a2d4e23a9c578727574a367" candidate_uuid: "bd85221f3ac548a29967a47c0a955ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de"
I20260620 06:58:12.095213 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dcecce26a2d4e23a9c578727574a367" candidate_uuid: "bd85221f3ac548a29967a47c0a955ee5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389b7db88f141fe9b406b6197844bdc"
I20260620 06:58:12.095383 7476 raft_consensus.cc:3037] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:12.095414 7622 raft_consensus.cc:3037] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:58:12.096392 7670 tablet.cc:2369] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:58:12.097918 7476 raft_consensus.cc:2454] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd85221f3ac548a29967a47c0a955ee5 in term 1.
I20260620 06:58:12.098306 7697 leader_election.cc:302] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d6d1c44f04db5ab57c8ca75c360de, bd85221f3ac548a29967a47c0a955ee5; no voters:
I20260620 06:58:12.098601 7852 raft_consensus.cc:2781] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:12.099395 7622 raft_consensus.cc:2454] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd85221f3ac548a29967a47c0a955ee5 in term 1.
I20260620 06:58:12.099788 7852 raft_consensus.cc:686] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 1 LEADER]: Becoming Leader. State: Replica: bd85221f3ac548a29967a47c0a955ee5, State: Running, Role: LEADER
I20260620 06:58:12.100281 7852 consensus_queue.cc:237] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:12.105041 6556 catalog_manager.cc:5495] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bd85221f3ac548a29967a47c0a955ee5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } health_report { overall_health: HEALTHY } } }
I20260620 06:58:12.188988 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.183841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.0.97.3:35001)
0620 06:58:12.184007 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.184013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.184032 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:58:12.184293 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.184297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.184373 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.184500 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:12.184869 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.185578 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.186986 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.187408 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.187514 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.187955 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:12.188446 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:12.188584 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.188668 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.188747 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:58:12.196949 7476 raft_consensus.cc:1261] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Refusing update from remote peer bd85221f3ac548a29967a47c0a955ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:12.196949 7622 raft_consensus.cc:1261] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Refusing update from remote peer bd85221f3ac548a29967a47c0a955ee5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:12.197922 7854 consensus_queue.cc:1035] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:12.198319 7852 consensus_queue.cc:1035] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:12.215191 7857 mvcc.cc:205] Tried to move back new op lower bound from 7298820883229274112 to 7298820882846126080. Current Snapshot: MvccSnapshot[applied={T|T < 7298820883229274112 or (T in {7298820883229274112})}]
I20260620 06:58:12.229127 6555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:12.229394 6555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:12.235529 6555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cfb2df0979b4700aabe071cf98f4838: Sending DeleteTablet for 3 replicas of tablet 8dcecce26a2d4e23a9c578727574a367
I20260620 06:58:12.236542 7602 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcecce26a2d4e23a9c578727574a367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57696
I20260620 06:58:12.236868 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcecce26a2d4e23a9c578727574a367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47424
I20260620 06:58:12.237233 7743 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dcecce26a2d4e23a9c578727574a367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:12 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47868
I20260620 06:58:12.238448 7866 tablet_replica.cc:330] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: stopping tablet replica
I20260620 06:58:12.238986 7866 raft_consensus.cc:2229] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:12.239316 7867 tablet_replica.cc:330] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: stopping tablet replica
I20260620 06:58:12.239671 7866 raft_consensus.cc:2258] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:12.239751 7867 raft_consensus.cc:2229] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:12.240248 7867 raft_consensus.cc:2258] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:12.241067 7868 tablet_replica.cc:330] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: stopping tablet replica
I20260620 06:58:12.241504 7868 raft_consensus.cc:2229] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:12.242208 7868 raft_consensus.cc:2258] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:12.244534 7867 ts_tablet_manager.cc:1905] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:12.244581 7868 ts_tablet_manager.cc:1905] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:12.247802 7866 ts_tablet_manager.cc:1905] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:12.251609 7867 ts_tablet_manager.cc:1918] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:12.251772 7867 log.cc:1196] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/8dcecce26a2d4e23a9c578727574a367
I20260620 06:58:12.252316 7867 ts_tablet_manager.cc:1939] T 8dcecce26a2d4e23a9c578727574a367 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting consensus metadata
I20260620 06:58:12.252894 7868 ts_tablet_manager.cc:1918] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:12.253146 7868 log.cc:1196] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/8dcecce26a2d4e23a9c578727574a367
I20260620 06:58:12.253752 7868 ts_tablet_manager.cc:1939] T 8dcecce26a2d4e23a9c578727574a367 P bd85221f3ac548a29967a47c0a955ee5: Deleting consensus metadata
I20260620 06:58:12.254161 6541 catalog_manager.cc:4841] TS 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147): tablet 8dcecce26a2d4e23a9c578727574a367 (table test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]) successfully deleted
I20260620 06:58:12.256035 6543 catalog_manager.cc:4841] TS bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001): tablet 8dcecce26a2d4e23a9c578727574a367 (table test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]) successfully deleted
I20260620 06:58:12.256511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 6524
I20260620 06:58:12.257659 6586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:12.260501 7866 ts_tablet_manager.cc:1918] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:12.260653 7866 log.cc:1196] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/8dcecce26a2d4e23a9c578727574a367
I20260620 06:58:12.261065 7866 ts_tablet_manager.cc:1939] T 8dcecce26a2d4e23a9c578727574a367 P d389b7db88f141fe9b406b6197844bdc: Deleting consensus metadata
I20260620 06:58:12.275560 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.240378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.0.97.61:32775)
0620 06:58:12.240592 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.240599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.255480 (+ 14881us) server_negotiation.cc:407] Connection header received
0620 06:58:12.255687 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.255693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.255796 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.255898 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:12.259155 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.260341 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.262170 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.262881 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.263011 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.267987 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:12.267994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:12.267999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:12.268046 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:12.272099 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.273027 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.273038 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.273043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.273203 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.273639 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.273645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.273649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.274040 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:12.274333 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.274926 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.275178 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:58:12.291389 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.255129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.97.60:44669)
0620 06:58:12.255336 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.255342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.279055 (+ 23713us) server_negotiation.cc:407] Connection header received
0620 06:58:12.279327 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.279333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.279429 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.279498 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:12.280906 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.281584 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.282868 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.283275 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.283386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.285791 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:12.285796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:12.285799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:12.285834 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:12.288308 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.288901 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.288906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.288908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.289030 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.289785 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.289790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.289793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.290055 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:12.290285 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.290624 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.291160 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:58:12.430156 6540 catalog_manager.cc:4841] TS d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257): tablet 8dcecce26a2d4e23a9c578727574a367 (table test-table [id=8ca929d46f72496ebe6c838cdcd5d4d7]) successfully deleted
I20260620 06:58:12.430279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 6524
I20260620 06:58:12.474864 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.451927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.0.97.60:44669)
0620 06:58:12.452093 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.452099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.459304 (+ 7205us) server_negotiation.cc:407] Connection header received
0620 06:58:12.459865 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.459869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.459942 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.460026 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:12.462126 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.462976 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.465134 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.465714 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.465838 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.469339 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:12.469346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:12.469350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:12.469391 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:12.471846 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.472479 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.472484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.472487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.472635 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.473318 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.473323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.473326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.473665 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:12.473872 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.474247 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.474465 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:58:12.475056 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:12.451994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.97.61:32775)
0620 06:58:12.452162 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:12.452183 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:12.458933 (+ 6750us) server_negotiation.cc:407] Connection header received
0620 06:58:12.459722 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:12.459729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:12.459830 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:12.459941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:12.461531 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.462436 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.464393 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:12.464806 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:12.464897 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:12.468727 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:12.468733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:12.468736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:12.468773 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:12.471837 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.473063 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.473069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.473073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.473197 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:12.473593 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:12.473598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:12.473601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:12.473945 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:12.474186 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:12.474508 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 06:58:12.474746 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:58:13.223424 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.222525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:54253)
0620 06:58:13.223074 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.223213 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":280,"threads_started":1}
W20260620 06:58:13.224017 7522 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.224197 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.223464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:56177)
0620 06:58:13.223884 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.224007 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260620 06:58:13.224429 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.223082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:59167)
0620 06:58:13.224137 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.224262 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":859,"threads_started":1}
W20260620 06:58:13.224511 7809 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
W20260620 06:58:13.224783 7668 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37021 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.735852 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9cfb2df0979b4700aabe071cf98f4838)
I20260620 06:58:13.736150 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } }
I20260620 06:58:13.736271 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9cfb2df0979b4700aabe071cf98f4838)
I20260620 06:58:13.736496 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } }
I20260620 06:58:13.737588 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021), cbc0fb9a2e8c4e1d904fff3fdc98bfd8 (127.0.97.60:44669)
I20260620 06:58:13.737744 7882 leader_election.cc:288] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021), 9feaf91cbec040869784ceec0cbab59f (127.0.97.61:32775)
I20260620 06:58:13.738859 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.737746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39654)
0620 06:58:13.738427 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.738607 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"threads_started":1}
I20260620 06:58:13.748178 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.738030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:32775 (local address 127.0.0.1:50638)
0620 06:58:13.739658 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.739674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.739779 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.740645 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.740650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.740679 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.740988 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.741001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.742753 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.742759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.745029 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.745045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.747324 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.747330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.747535 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.747567 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.747651 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.747776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"thread_start_us":1386,"threads_started":1}
I20260620 06:58:13.748781 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.738381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39660)
0620 06:58:13.748489 (+ 10108us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.748637 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9956,"threads_started":1}
I20260620 06:58:13.749426 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.738491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.0.97.61:32775)
0620 06:58:13.740192 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.740199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.740217 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:58:13.740281 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.740287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.740357 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.740448 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.741189 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.742581 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.745230 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.747012 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.747136 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.747223 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.748926 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.749068 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1604,"thread_start_us":1425,"threads_started":1}
I20260620 06:58:13.749965 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9feaf91cbec040869784ceec0cbab59f" is_pre_election: true
W20260620 06:58:13.750030 7326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.750304 7163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbc0fb9a2e8c4e1d904fff3fdc98bfd8 in term 1.
I20260620 06:58:13.750934 7329 leader_election.cc:302] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9feaf91cbec040869784ceec0cbab59f, cbc0fb9a2e8c4e1d904fff3fdc98bfd8; no voters:
I20260620 06:58:13.751401 7882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260620 06:58:13.751336 7137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.751539 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9cfb2df0979b4700aabe071cf98f4838)
I20260620 06:58:13.751657 7882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:58:13.752533 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.751954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39670)
0620 06:58:13.752111 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.752280 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:58:13.752751 7326 leader_election.cc:334] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.753578 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.753095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39676)
0620 06:58:13.753292 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.753425 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 06:58:13.753753 7137 leader_election.cc:334] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.754534 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } }
I20260620 06:58:13.755155 7882 leader_election.cc:288] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 election: Requested vote from peers 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021), 9feaf91cbec040869784ceec0cbab59f (127.0.97.61:32775)
I20260620 06:58:13.755590 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.755238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39682)
0620 06:58:13.755377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.755451 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 06:58:13.755553 7163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9feaf91cbec040869784ceec0cbab59f"
I20260620 06:58:13.755746 7163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:58:13.756569 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.756180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39690)
0620 06:58:13.756323 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.756405 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:58:13.756748 7326 leader_election.cc:334] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021): Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
I20260620 06:58:13.757555 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.737819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44669 (local address 127.0.0.1:56742)
0620 06:58:13.739036 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.739053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.739196 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.750283 (+ 11087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.750288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.750319 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.750594 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.750605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.753824 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.753833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.755296 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.755310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.756806 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.756813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.756980 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.757012 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.757089 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.757235 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":456,"threads_started":1}
I20260620 06:58:13.757555 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.738171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.0.97.60:44669)
0620 06:58:13.749979 (+ 11808us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.749985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.750002 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:58:13.750058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.750062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.750129 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.750717 (+ 588us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.752182 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.753627 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.755427 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.756571 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.756679 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.756783 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.757140 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.757258 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11678,"thread_start_us":12861,"threads_started":1}
I20260620 06:58:13.758121 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9feaf91cbec040869784ceec0cbab59f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" is_pre_election: true
I20260620 06:58:13.758275 7163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc0fb9a2e8c4e1d904fff3fdc98bfd8 in term 2.
I20260620 06:58:13.758467 7352 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9feaf91cbec040869784ceec0cbab59f in current term 2: Already voted for candidate cbc0fb9a2e8c4e1d904fff3fdc98bfd8 in this term.
I20260620 06:58:13.758659 7329 leader_election.cc:302] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9feaf91cbec040869784ceec0cbab59f, cbc0fb9a2e8c4e1d904fff3fdc98bfd8; no voters: 9cfb2df0979b4700aabe071cf98f4838
I20260620 06:58:13.758948 7882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:58:13.759078 7139 leader_election.cc:302] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9feaf91cbec040869784ceec0cbab59f; no voters: 9cfb2df0979b4700aabe071cf98f4838, cbc0fb9a2e8c4e1d904fff3fdc98bfd8
I20260620 06:58:13.759354 7882 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [term 2 LEADER]: Becoming Leader. State: Replica: cbc0fb9a2e8c4e1d904fff3fdc98bfd8, State: Running, Role: LEADER
I20260620 06:58:13.759436 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:58:13.759747 7882 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } }
I20260620 06:58:13.761492 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc0fb9a2e8c4e1d904fff3fdc98bfd8. Latest consensus state: current_term: 2 leader_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:13.761845 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:13.762416 7893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:13.765820 7893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:13.766624 7893 catalog_manager.cc:1259] Loaded cluster ID: afdf28a77a0a41679c5939805be3bde4
I20260620 06:58:13.766688 7893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:13.767786 7893 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:13.768493 7893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Loaded TSK: 0
I20260620 06:58:13.769286 7893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:58:13.802975 7342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:13.809765 7342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:13.813616 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.813231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39708)
0620 06:58:13.813354 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.813437 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 06:58:13.813824 7163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [term 2 FOLLOWER]: Refusing update from remote peer cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 06:58:13.814687 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.814223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39718)
0620 06:58:13.814386 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.814519 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:58:13.814597 7891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 06:58:13.814862 7326 consensus_peers.cc:487] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 -> Peer 9cfb2df0979b4700aabe071cf98f4838 (127.0.97.62:37021): Couldn't send request to peer 9cfb2df0979b4700aabe071cf98f4838. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:58:13.817574 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc0fb9a2e8c4e1d904fff3fdc98bfd8. Latest consensus state: current_term: 2 leader_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:13.817857 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:13.818643 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:13.818897 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:13.819419 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9cfb2df0979b4700aabe071cf98f4838" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37021 } } peers { permanent_uuid: "9feaf91cbec040869784ceec0cbab59f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "cbc0fb9a2e8c4e1d904fff3fdc98bfd8" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44669 } attrs { promote: false } } }
I20260620 06:58:13.819675 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9feaf91cbec040869784ceec0cbab59f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:58:13.834767 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.826957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47430)
0620 06:58:13.827088 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.827104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.827205 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.828206 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.828211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.828240 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.828578 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.828589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.830056 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.830061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.832301 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.832319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.833931 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.833939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.834139 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.834179 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.834248 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.834407 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 06:58:13.836020 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.827437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.97.1:41147)
0620 06:58:13.827876 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.827881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.827906 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:58:13.827969 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.827974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.828041 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.828714 (+ 673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.828861 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.829922 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.832535 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.833694 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.835375 (+ 1681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.835491 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.835543 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.835687 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":220,"threads_started":1}
I20260620 06:58:13.836813 7456 tablet_service.cc:1467] Processing CreateTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 (DEFAULT_TABLE table=test-table [id=660cade3894d4af69d42e8fdc10c83ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:13.837278 7456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f935c73e5b24c3da4f8069b9b509c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:13.844331 7904 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Bootstrap starting.
I20260620 06:58:13.844933 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.827742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35001 (local address 127.0.0.1:47880)
0620 06:58:13.827882 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.827902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.827992 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.839383 (+ 11391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.839388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.839423 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.839563 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.839573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.840973 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.840978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.842785 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.842801 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.844209 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.844215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.844414 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.844447 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.844512 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.844643 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 06:58:13.845620 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.828034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.0.97.3:35001)
0620 06:58:13.839033 (+ 10999us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.839040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.839061 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:58:13.839135 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.839141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.839231 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.839352 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.839697 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.840788 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.842959 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.844067 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.845148 (+ 1081us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.845246 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.845312 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.845442 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10870,"thread_start_us":10772,"threads_started":1}
I20260620 06:58:13.846266 7743 tablet_service.cc:1467] Processing CreateTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 (DEFAULT_TABLE table=test-table [id=660cade3894d4af69d42e8fdc10c83ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:13.846650 7743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f935c73e5b24c3da4f8069b9b509c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:13.846649 7904 tablet_bootstrap.cc:654] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:13.848557 7904 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: No bootstrap required, opened a new log
I20260620 06:58:13.848688 7904 ts_tablet_manager.cc:1397] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260620 06:58:13.849316 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.827846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46257 (local address 127.0.0.1:57706)
0620 06:58:13.835005 (+ 7159us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.835023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.835128 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.843811 (+ 8683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.843816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.843852 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.843999 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.844009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.845378 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.845383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.847070 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.847085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.848538 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.848544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.848726 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.848759 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.848846 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.849000 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7065,"thread_start_us":10683,"threads_started":1}
I20260620 06:58:13.849706 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.828209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.97.2:46257)
0620 06:58:13.843413 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.843419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.843437 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:58:13.843499 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.843504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.843576 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.844135 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.844287 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.845245 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.847242 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.848362 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.848526 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.848621 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.849304 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.849438 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15084,"thread_start_us":15442,"threads_started":1}
I20260620 06:58:13.850145 7904 raft_consensus.cc:348] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.850389 7904 raft_consensus.cc:374] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:13.850338 7602 tablet_service.cc:1467] Processing CreateTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 (DEFAULT_TABLE table=test-table [id=660cade3894d4af69d42e8fdc10c83ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:13.850488 7904 raft_consensus.cc:729] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4d6d1c44f04db5ab57c8ca75c360de, State: Initialized, Role: FOLLOWER
I20260620 06:58:13.850746 7602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f935c73e5b24c3da4f8069b9b509c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:13.850741 7904 consensus_queue.cc:260] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.851501 7904 ts_tablet_manager.cc:1428] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 06:58:13.854663 7521 heartbeater.cc:503] Master 127.0.97.60:44669 was elected leader, sending a full tablet report...
I20260620 06:58:13.854813 7908 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Bootstrap starting.
I20260620 06:58:13.856360 7908 tablet_bootstrap.cc:654] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:13.857117 7909 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Bootstrap starting.
I20260620 06:58:13.858711 7909 tablet_bootstrap.cc:654] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:13.859532 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.852032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:47409)
0620 06:58:13.859118 (+ 7086us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.859261 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6935,"thread_start_us":6794,"threads_started":1}
I20260620 06:58:13.860025 7908 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: No bootstrap required, opened a new log
I20260620 06:58:13.860283 7908 ts_tablet_manager.cc:1397] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:58:13.860836 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.860489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:48857)
0620 06:58:13.860621 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.860703 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 06:58:13.861053 7908 raft_consensus.cc:348] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.861230 7908 raft_consensus.cc:374] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:13.861315 7908 raft_consensus.cc:729] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd85221f3ac548a29967a47c0a955ee5, State: Initialized, Role: FOLLOWER
I20260620 06:58:13.861529 7908 consensus_queue.cc:260] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.861806 7909 tablet_bootstrap.cc:492] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: No bootstrap required, opened a new log
I20260620 06:58:13.861946 7909 ts_tablet_manager.cc:1397] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:58:13.862243 7908 ts_tablet_manager.cc:1428] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:58:13.862711 7909 raft_consensus.cc:348] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.862890 7909 raft_consensus.cc:374] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:13.862979 7909 raft_consensus.cc:729] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d389b7db88f141fe9b406b6197844bdc, State: Initialized, Role: FOLLOWER
I20260620 06:58:13.863195 7909 consensus_queue.cc:260] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.864328 7808 heartbeater.cc:503] Master 127.0.97.60:44669 was elected leader, sending a full tablet report...
I20260620 06:58:13.867416 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.862466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:52003)
0620 06:58:13.867087 (+ 4621us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.867179 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4515,"thread_start_us":4359,"threads_started":1}
I20260620 06:58:13.868084 7909 ts_tablet_manager.cc:1428] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260620 06:58:13.868774 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.868219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:35833)
0620 06:58:13.868495 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.868582 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20260620 06:58:13.869896 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.868219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:32933)
0620 06:58:13.869594 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.869699 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1253,"thread_start_us":1138,"threads_started":1}
I20260620 06:58:13.870339 7667 heartbeater.cc:503] Master 127.0.97.60:44669 was elected leader, sending a full tablet report...
I20260620 06:58:13.870973 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.870583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:58227)
0620 06:58:13.870771 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.870854 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 06:58:13.972324 7906 raft_consensus.cc:482] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:13.972566 7906 raft_consensus.cc:504] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.974040 7906 leader_election.cc:288] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257), bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:13.979875 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.974424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32775 (local address 127.0.97.2:46257)
0620 06:58:13.974705 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.974711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.974821 (+ 110us) server_negotiation.cc:407] Connection header received
0620 06:58:13.975075 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.975081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.975153 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.975228 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.975674 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.976360 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.978096 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.978976 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.979102 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.979199 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.979518 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.979641 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 06:58:13.979909 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.974421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43663 (local address 127.0.97.3:35001)
0620 06:58:13.974649 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:13.974656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:13.975037 (+ 381us) server_negotiation.cc:407] Connection header received
0620 06:58:13.975190 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:13.975196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:13.975254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:13.975325 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:13.975806 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.976474 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.978115 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.979059 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.979172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.979260 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:13.979576 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 06:58:13.979697 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:58:13.979904 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.974315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46257 (local address 127.0.97.1:32775)
0620 06:58:13.974718 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.974740 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.974939 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.975264 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.975268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.975291 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.975547 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.975555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.976484 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.976489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.977947 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.977961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.979156 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.979162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.979349 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.979381 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.979472 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.979601 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":1}
I20260620 06:58:13.980484 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.974315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35001 (local address 127.0.97.1:43663)
0620 06:58:13.974925 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.974938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:13.975064 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:13.975345 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:13.975349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:13.975375 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:13.975676 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.975684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.976602 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.976605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.977974 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:13.977985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:13.979251 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:13.979257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:13.979402 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:13.979423 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:58:13.980072 (+ 649us) client_negotiation.cc:240] Negotiation successful
0620 06:58:13.980188 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":96,"thread_start_us":592,"threads_started":1}
I20260620 06:58:13.980508 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f935c73e5b24c3da4f8069b9b509c97" candidate_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389b7db88f141fe9b406b6197844bdc" is_pre_election: true
I20260620 06:58:13.980764 7622 raft_consensus.cc:2454] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d6d1c44f04db5ab57c8ca75c360de in term 0.
I20260620 06:58:13.981230 7409 leader_election.cc:302] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d6d1c44f04db5ab57c8ca75c360de, d389b7db88f141fe9b406b6197844bdc; no voters:
I20260620 06:58:13.981240 7763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f935c73e5b24c3da4f8069b9b509c97" candidate_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd85221f3ac548a29967a47c0a955ee5" is_pre_election: true
I20260620 06:58:13.981545 7906 raft_consensus.cc:2781] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:13.981593 7763 raft_consensus.cc:2454] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4d6d1c44f04db5ab57c8ca75c360de in term 0.
I20260620 06:58:13.981642 7906 raft_consensus.cc:482] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:13.981725 7906 raft_consensus.cc:3037] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:13.983950 7906 raft_consensus.cc:504] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.984512 7906 leader_election.cc:288] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [CANDIDATE]: Term 1 election: Requested vote from peers d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257), bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001)
I20260620 06:58:13.984910 7622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f935c73e5b24c3da4f8069b9b509c97" candidate_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d389b7db88f141fe9b406b6197844bdc"
I20260620 06:58:13.984998 7763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f935c73e5b24c3da4f8069b9b509c97" candidate_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd85221f3ac548a29967a47c0a955ee5"
I20260620 06:58:13.985080 7622 raft_consensus.cc:3037] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:13.985178 7763 raft_consensus.cc:3037] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:13.987169 7622 raft_consensus.cc:2454] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d6d1c44f04db5ab57c8ca75c360de in term 1.
I20260620 06:58:13.987442 7763 raft_consensus.cc:2454] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4d6d1c44f04db5ab57c8ca75c360de in term 1.
I20260620 06:58:13.987490 7409 leader_election.cc:302] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4d6d1c44f04db5ab57c8ca75c360de, d389b7db88f141fe9b406b6197844bdc; no voters:
I20260620 06:58:13.987787 7906 raft_consensus.cc:2781] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:13.988214 7906 raft_consensus.cc:686] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 LEADER]: Becoming Leader. State: Replica: 2e4d6d1c44f04db5ab57c8ca75c360de, State: Running, Role: LEADER
I20260620 06:58:13.988601 7906 consensus_queue.cc:237] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } }
I20260620 06:58:13.989239 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:13.988752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:42879)
0620 06:58:13.988951 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:13.989077 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 06:58:13.992038 7342 catalog_manager.cc:5495] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4d6d1c44f04db5ab57c8ca75c360de" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 } health_report { overall_health: UNKNOWN } } }
I20260620 06:58:14.028366 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.022837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.1:41147)
0620 06:58:14.023028 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:14.023033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:14.023052 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:58:14.023266 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:14.023271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:14.023363 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:14.023475 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:14.023822 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.024471 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.026324 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.026798 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.026888 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:14.027349 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:14.027883 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:14.027991 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:14.028078 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 06:58:14.028138 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:58:14.032379 7622 raft_consensus.cc:1261] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Refusing update from remote peer 2e4d6d1c44f04db5ab57c8ca75c360de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:14.032625 7763 raft_consensus.cc:1261] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Refusing update from remote peer 2e4d6d1c44f04db5ab57c8ca75c360de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:14.033118 7917 consensus_queue.cc:1035] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [LEADER]: Connected to new peer: Peer: permanent_uuid: "d389b7db88f141fe9b406b6197844bdc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:14.033505 7906 consensus_queue.cc:1035] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd85221f3ac548a29967a47c0a955ee5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:14.035806 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.035407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:46649)
0620 06:58:14.035539 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.035639 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 06:58:14.036340 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.035609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:36251)
0620 06:58:14.036052 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.036163 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
I20260620 06:58:14.038740 7920 mvcc.cc:205] Tried to move back new op lower bound from 7298820890749550592 to 7298820890581168128. Current Snapshot: MvccSnapshot[applied={T|T < 7298820890749550592}]
I20260620 06:58:14.041072 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.040587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.3:33267)
0620 06:58:14.040797 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.040894 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 06:58:14.042757 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.042215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:46613)
0620 06:58:14.042480 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.042594 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 06:58:14.051923 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.051446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.1:56195)
0620 06:58:14.051647 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.051729 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 06:58:14.052851 7921 mvcc.cc:205] Tried to move back new op lower bound from 7298820890749550592 to 7298820890581168128. Current Snapshot: MvccSnapshot[applied={T|T < 7298820890749550592}]
I20260620 06:58:14.053982 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.053034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.97.2:58103)
0620 06:58:14.053749 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.053840 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614}
I20260620 06:58:14.055788 7341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:14.056102 7341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:14.062971 7341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbc0fb9a2e8c4e1d904fff3fdc98bfd8: Sending DeleteTablet for 3 replicas of tablet 3f935c73e5b24c3da4f8069b9b509c97
I20260620 06:58:14.063901 7602 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57706
I20260620 06:58:14.064183 7743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47880
I20260620 06:58:14.064472 7456 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f935c73e5b24c3da4f8069b9b509c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:14 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:47430
I20260620 06:58:14.066339 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7379
I20260620 06:58:14.095270 7932 tablet_replica.cc:330] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: stopping tablet replica
I20260620 06:58:14.095655 7932 raft_consensus.cc:2229] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:14.098709 7932 raft_consensus.cc:2258] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:14.099967 7932 ts_tablet_manager.cc:1905] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:14.111261 7930 tablet_replica.cc:330] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: stopping tablet replica
I20260620 06:58:14.111788 7931 tablet_replica.cc:330] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: stopping tablet replica
I20260620 06:58:14.112061 7931 raft_consensus.cc:2229] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:14.112327 7930 raft_consensus.cc:2229] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:14.112687 7931 raft_consensus.cc:2258] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:14.113019 7930 raft_consensus.cc:2258] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:14.116351 7932 ts_tablet_manager.cc:1918] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:14.116930 7931 ts_tablet_manager.cc:1905] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:14.117319 7930 ts_tablet_manager.cc:1905] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:14.123225 7931 ts_tablet_manager.cc:1918] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:14.123078 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.068086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.0.97.1:41147)
0620 06:58:14.068258 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:14.068264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:14.094808 (+ 26544us) server_negotiation.cc:407] Connection header received
0620 06:58:14.096184 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:14.096191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:14.096274 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:14.096349 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:14.102672 (+ 6323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.103821 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.105855 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.106555 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.106656 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:14.115854 (+ 9198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:14.115869 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:14.115888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:14.115940 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:14.119703 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.120508 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.120517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.120521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.120728 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.121199 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.121204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.121208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.121638 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:14.121914 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:14.122462 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 06:58:14.122700 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:58:14.123469 7930 ts_tablet_manager.cc:1918] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:14.123589 7930 log.cc:1196] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/3f935c73e5b24c3da4f8069b9b509c97
I20260620 06:58:14.123970 7930 ts_tablet_manager.cc:1939] T 3f935c73e5b24c3da4f8069b9b509c97 P d389b7db88f141fe9b406b6197844bdc: Deleting consensus metadata
I20260620 06:58:14.124338 7516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:14.124883 7932 log.cc:1196] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/3f935c73e5b24c3da4f8069b9b509c97
I20260620 06:58:14.124940 7326 catalog_manager.cc:4841] TS d389b7db88f141fe9b406b6197844bdc (127.0.97.2:46257): tablet 3f935c73e5b24c3da4f8069b9b509c97 (table test-table [id=660cade3894d4af69d42e8fdc10c83ac]) successfully deleted
I20260620 06:58:14.125046 7931 log.cc:1196] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/3f935c73e5b24c3da4f8069b9b509c97
I20260620 06:58:14.125473 7931 ts_tablet_manager.cc:1939] T 3f935c73e5b24c3da4f8069b9b509c97 P bd85221f3ac548a29967a47c0a955ee5: Deleting consensus metadata
I20260620 06:58:14.126564 7329 catalog_manager.cc:4841] TS bd85221f3ac548a29967a47c0a955ee5 (127.0.97.3:35001): tablet 3f935c73e5b24c3da4f8069b9b509c97 (table test-table [id=660cade3894d4af69d42e8fdc10c83ac]) successfully deleted
I20260620 06:58:14.227025 7932 ts_tablet_manager.cc:1939] T 3f935c73e5b24c3da4f8069b9b509c97 P 2e4d6d1c44f04db5ab57c8ca75c360de: Deleting consensus metadata
I20260620 06:58:14.227376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7379
I20260620 06:58:14.242411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7533
I20260620 06:58:14.242698 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.242237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47450)
0620 06:58:14.242421 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.242506 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 06:58:14.243245 7327 catalog_manager.cc:4584] TS 2e4d6d1c44f04db5ab57c8ca75c360de (127.0.97.1:41147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f935c73e5b24c3da4f8069b9b509c97: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
I20260620 06:58:14.255344 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.243170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.0.97.2:46257)
0620 06:58:14.243654 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:14.243661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:14.243682 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:58:14.243747 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:14.243752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:14.243841 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:14.243953 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:14.245106 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.245950 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.247392 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.247761 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.247852 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:14.250059 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:14.250074 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:14.250092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:14.250124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:14.252716 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.253377 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.253385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.253387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.253503 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.253848 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.253853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.253856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.254206 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:14.254483 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:14.254812 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 06:58:14.255046 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260620 06:58:14.255925 7662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:14.303957 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.303388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47460)
0620 06:58:14.303643 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.303765 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 06:58:14.305002 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.304631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47474)
0620 06:58:14.304781 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.304865 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:58:14.347797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7533
I20260620 06:58:14.351130 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.350698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39720)
0620 06:58:14.350899 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.350998 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:58:14.352102 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.351744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37021 (local address 127.0.0.1:39728)
0620 06:58:14.351895 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.351972 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 06:58:14.362975 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7680
I20260620 06:58:14.373788 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.373314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47480)
0620 06:58:14.373534 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.373630 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 06:58:14.375038 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.374477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47486)
0620 06:58:14.374852 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.374923 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 06:58:14.376174 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.363666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.0.97.3:35001)
0620 06:58:14.363829 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:14.363836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:14.363896 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:58:14.364081 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:14.364087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:14.364154 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:14.364235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:14.365539 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.366272 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.367503 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:14.367883 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:14.367973 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:14.370280 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:14.370294 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:14.370310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:14.370348 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:14.373486 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.374242 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.374253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.374256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.374469 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:14.374848 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:14.374852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:14.374854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:14.375243 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:14.375507 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:14.375725 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:58:14.375879 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:58:14.376914 7803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:14.476652 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.476148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47488)
0620 06:58:14.476344 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.476442 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 06:58:14.477630 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.477301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47500)
0620 06:58:14.477448 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.477522 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:58:14.490669 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7680
I20260620 06:58:14.507840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7017
I20260620 06:58:14.508800 7183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:14.621521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7017
I20260620 06:58:14.637722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7116
I20260620 06:58:14.638733 7372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:14.753782 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:14.753337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41147 (local address 127.0.0.1:47516)
0620 06:58:14.753541 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:14.753624 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 06:58:14.754041 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7116
2026-06-20T06:58:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9530 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:14 dist-test-slave-mz20 krb5kdc[7941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:14 dist-test-slave-mz20 krb5kdc[7941](info): set up 2 sockets
Jun 20 06:58:14 dist-test-slave-mz20 krb5kdc[7941](info): commencing operation
krb5kdc: starting...
W20260620 06:58:15.759101 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.006s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:15 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 06:58:15.956269 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34305
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:16.262297 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:16.262626 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:16.262763 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:16.269867 7954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:16.269989 7954 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:58:16.270056 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:16.270138 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:16.270202 7954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:16.270282 7954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:16.280164 7954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:16.281479 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:16.282843 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:16.294262 7960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:16.294298 7962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:16.294348 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:16.295161 7954 server_base.cc:1034] running on GCE node
I20260620 06:58:16.295892 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:16.297214 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:16.298418 7954 hybrid_clock.cc:648] HybridClock initialized: now 1781938696298358 us; error 34 us; skew 500 ppm
I20260620 06:58:16.298746 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:16 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:16.301640 7954 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 06:58:16.303054 7954 webserver.cc:460] Webserver started at http://127.0.97.62:44791/ using document root <none> and password file <none>
I20260620 06:58:16.303738 7954 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:16.303833 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:16.304131 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:16.306969 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "b92ce1730d47430aa9321f0446d8059d"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "846d9052e4ddeedf177d5a05b905175c"
server_key_iv: "055c7fef29a28bbb38bf3b5839bbb6d9"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.307706 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "b92ce1730d47430aa9321f0446d8059d"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "846d9052e4ddeedf177d5a05b905175c"
server_key_iv: "055c7fef29a28bbb38bf3b5839bbb6d9"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.312500 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:58:16.315799 7969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:16.317075 7954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:16.317231 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "b92ce1730d47430aa9321f0446d8059d"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "846d9052e4ddeedf177d5a05b905175c"
server_key_iv: "055c7fef29a28bbb38bf3b5839bbb6d9"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.317468 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:16.350212 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:16.354091 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:16.354324 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:16.374877 7954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34305
I20260620 06:58:16.374895 8030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34305 every 8 connection(s)
I20260620 06:58:16.376134 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:16.379698 8031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:16.385051 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 7954
I20260620 06:58:16.385269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:16.385998 388 external_mini_cluster.cc:1428] Setting key ae47ba78cef7c4f53d57702f932f3d76
I20260620 06:58:16.390630 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Bootstrap starting.
I20260620 06:58:16.394048 8031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:16.395324 8031 log.cc:824] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:16.398510 8031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: No bootstrap required, opened a new log
Jun 20 06:58:16 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:16.407171 8031 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } }
I20260620 06:58:16.407557 8031 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:16.407656 8031 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92ce1730d47430aa9321f0446d8059d, State: Initialized, Role: FOLLOWER
I20260620 06:58:16.408217 8031 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } }
I20260620 06:58:16.408442 8031 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:16.408565 8031 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:16.408717 8031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:16.408748 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:16.388040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.0.97.62:34305)
0620 06:58:16.388848 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:16.388860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:16.388901 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:58:16.390502 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:16.390525 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:16.390983 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:16.391300 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:16.392921 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.394237 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.395924 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.396859 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.396986 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:16.401448 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:16.401456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:16.401473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:16.401514 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:16.404418 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.405200 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.405206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.405213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.405378 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.405722 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.405725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.405727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.406247 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:16.406466 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:16.407024 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 06:58:16.407550 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":385,"threads_started":1}
I20260620 06:58:16.411509 8031 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } }
I20260620 06:58:16.412009 8031 leader_election.cc:302] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92ce1730d47430aa9321f0446d8059d; no voters:
I20260620 06:58:16.412537 8031 leader_election.cc:288] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:16.412649 8036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:16.413302 8036 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [term 1 LEADER]: Becoming Leader. State: Replica: b92ce1730d47430aa9321f0446d8059d, State: Running, Role: LEADER
I20260620 06:58:16.413818 8036 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } }
I20260620 06:58:16.414563 8031 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:16.416467 8037 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92ce1730d47430aa9321f0446d8059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } } }
I20260620 06:58:16.416524 8038 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92ce1730d47430aa9321f0446d8059d. Latest consensus state: current_term: 1 leader_uuid: "b92ce1730d47430aa9321f0446d8059d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92ce1730d47430aa9321f0446d8059d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34305 } } }
I20260620 06:58:16.416877 8038 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:16.416875 8037 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:16.419562 8043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:16.424526 8043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:16.432921 8043 catalog_manager.cc:1347] Generated new cluster ID: ed8c64cc8dae4163b9d2432a1165fa88
I20260620 06:58:16.433023 8043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:16.449967 8043 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:16.451318 8043 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:16.457965 8043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Generated new TSK 0
I20260620 06:58:16.458828 8043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 06:58:16.509711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34305
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:16.826853 8057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:16.827119 8057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:16.827220 8057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:16.834681 8057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:16.834810 8057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:16.834903 8057 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:58:16.834982 8057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:16.844657 8057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:16.845876 8057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:16.847252 8057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:16.859380 8065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:16.859421 8063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:16.859479 8062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:16.860055 8057 server_base.cc:1034] running on GCE node
I20260620 06:58:16.860692 8057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:16.861515 8057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:16.862717 8057 hybrid_clock.cc:648] HybridClock initialized: now 1781938696862657 us; error 40 us; skew 500 ppm
I20260620 06:58:16.862993 8057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:16 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:16.865870 8057 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 06:58:16.867214 8057 webserver.cc:460] Webserver started at http://127.0.97.1:37677/ using document root <none> and password file <none>
I20260620 06:58:16.867844 8057 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:16.867933 8057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:16.868180 8057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:16.870940 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "c168093cb7d5487eacc71ee1d8124460"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "67b1da5a6a1edb5afe4c4b7a1d8da3fe"
server_key_iv: "19a69e5a5f5d2068dc00d9e317729904"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.871630 8057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c168093cb7d5487eacc71ee1d8124460"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "67b1da5a6a1edb5afe4c4b7a1d8da3fe"
server_key_iv: "19a69e5a5f5d2068dc00d9e317729904"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.876314 8057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 06:58:16.879622 8072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:16.881280 8057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:58:16.881448 8057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "c168093cb7d5487eacc71ee1d8124460"
format_stamp: "Formatted at 2026-06-20 06:58:16 on dist-test-slave-mz20"
server_key: "67b1da5a6a1edb5afe4c4b7a1d8da3fe"
server_key_iv: "19a69e5a5f5d2068dc00d9e317729904"
server_key_version: "encryptionkey@0"
I20260620 06:58:16.881698 8057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:16.897583 8057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:16.901161 8057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:16.901396 8057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:16.903854 8057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:16.903934 8057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:16.904058 8057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:16.904114 8057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:58:16 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:16.926113 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:16.904490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57167 (local address 127.0.97.62:34305)
0620 06:58:16.904670 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:16.904677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:16.905942 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 06:58:16.908129 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:16.908135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:16.908220 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:16.908326 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:16.910465 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.911427 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.914174 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.914772 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.914900 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:16.919407 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:16.919412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:16.919416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:16.919453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:16.922576 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.923515 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.923521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.923526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.923680 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.924106 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.924111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.924114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.924455 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:16.924591 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:16.925591 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 06:58:16.925820 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:58:16.927878 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:16.904977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.1:57167)
0620 06:58:16.905766 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:16.905818 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:16.907825 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:16.908701 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:16.908711 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:16.909438 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:16.910258 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:16.910281 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.911607 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.911617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:16.914012 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:16.914028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.915020 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.915027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:16.915337 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:16.916207 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:16.916228 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:16.919167 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:16.922725 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:16.922731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:16.922747 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:16.923335 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:16.923811 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:16.923816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:16.923820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:16.924002 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:16.924627 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:16.924639 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:16.925136 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 06:58:16.925499 (+ 363us) client_negotiation.cc:240] Negotiation successful
0620 06:58:16.926115 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260620 06:58:16.939544 8057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39045
I20260620 06:58:16.939613 8191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39045 every 8 connection(s)
I20260620 06:58:16.940798 8057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:16.942827 8192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34305
I20260620 06:58:16.943038 8192 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:16.943643 8192 heartbeater.cc:511] Master 127.0.97.62:34305 requested a full tablet report, sending...
I20260620 06:58:16.945611 7986 ts_manager.cc:194] Registered new tserver with Master: c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045)
I20260620 06:58:16.947113 7986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57167
I20260620 06:58:16.948310 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:16.935337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40689 (local address 127.0.97.62:34305)
0620 06:58:16.935519 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:16.935525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:16.936293 (+ 768us) server_negotiation.cc:407] Connection header received
0620 06:58:16.936431 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:16.936437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:16.936519 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:16.936623 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:16.938008 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.939132 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.940584 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.941184 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.941323 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:16.943415 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:16.943421 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:16.943425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:16.943455 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:16.945925 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.946504 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.946510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.946513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.946673 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:16.946993 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:16.946998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:16.947001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:16.947297 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:16.947426 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:16.947774 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 06:58:16.947965 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:58:16.948395 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:16.935394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.1:40689)
0620 06:58:16.936181 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:16.936205 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:16.936328 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:16.936672 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:16.936678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:16.937399 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:16.937830 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:16.937844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.939248 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.939253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:16.940452 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:16.940464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:16.941386 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:16.941393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:16.941584 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:16.942284 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:16.942313 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:16.943254 (+ 941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:16.946042 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:16.946048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:16.946053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:16.946396 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:16.946776 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:16.946781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:16.946785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:16.946899 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:16.947397 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:16.947401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:16.947560 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 06:58:16.947672 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 06:58:16.947971 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":525,"threads_started":1}
I20260620 06:58:16.948695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8057
I20260620 06:58:16.948902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:16.949644 388 external_mini_cluster.cc:1428] Setting key 4d9bf0704034f170d466615037a789d4
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 06:58:16.992049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34305
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:17.304888 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:17.305167 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:17.305303 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:17.312493 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:17.312611 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:17.312721 8196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:58:17.312822 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:17.322788 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:17.324039 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:17.325368 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:17.337899 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:17.337872 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:17.337939 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:17.338774 8196 server_base.cc:1034] running on GCE node
I20260620 06:58:17.339375 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:17.340281 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:17.341488 8196 hybrid_clock.cc:648] HybridClock initialized: now 1781938697341417 us; error 43 us; skew 500 ppm
I20260620 06:58:17.341761 8196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:17.344734 8196 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 06:58:17.346107 8196 webserver.cc:460] Webserver started at http://127.0.97.2:45311/ using document root <none> and password file <none>
I20260620 06:58:17.346798 8196 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:17.346896 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:17.347193 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:17.350459 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "96f17b9a21ad165accf1e529590602fc"
server_key_iv: "6956aa8b24c39d931b69f7ad9239328a"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.351475 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "96f17b9a21ad165accf1e529590602fc"
server_key_iv: "6956aa8b24c39d931b69f7ad9239328a"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.357497 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 06:58:17.360975 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:17.362406 8196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:58:17.362594 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "96f17b9a21ad165accf1e529590602fc"
server_key_iv: "6956aa8b24c39d931b69f7ad9239328a"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.362799 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:17.376187 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:17.380120 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:17.380343 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:17.382879 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:17.382968 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:17.383083 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:17.383153 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:17.408417 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.383569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58293 (local address 127.0.97.62:34305)
0620 06:58:17.383745 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.383751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.385829 (+ 2078us) server_negotiation.cc:407] Connection header received
0620 06:58:17.388954 (+ 3125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.388960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.389057 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.389137 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.392015 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.392906 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.395712 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.396362 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.396504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.401431 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:17.401437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:17.401443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:17.401493 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:17.404860 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.405663 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.405670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.405675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.405817 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.406309 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.406314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.406318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.406709 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:17.406859 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:17.407836 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 06:58:17.408080 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:58:17.410082 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.384604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.2:58293)
0620 06:58:17.385583 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:17.385643 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:17.388553 (+ 2910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:17.389709 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:17.389722 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:17.390604 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:17.391775 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.391809 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.393096 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.393103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.395513 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.395540 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.396838 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.396846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.397166 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.398033 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:17.398066 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:17.401172 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:17.405002 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.405011 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.405029 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.405538 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.405924 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.405929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.405933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.406190 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.406866 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:17.406875 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:17.407389 (+ 514us) client_negotiation.cc:769] Sending connection context
0620 06:58:17.407734 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 06:58:17.408327 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":328,"threads_started":1}
I20260620 06:58:17.425165 8196 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41361
I20260620 06:58:17.425181 8330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41361 every 8 connection(s)
I20260620 06:58:17.426712 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:17.428970 8331 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34305
I20260620 06:58:17.429165 8331 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:17.429806 8331 heartbeater.cc:511] Master 127.0.97.62:34305 requested a full tablet report, sending...
I20260620 06:58:17.430413 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.416100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47977 (local address 127.0.97.62:34305)
0620 06:58:17.416308 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.416315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.416864 (+ 549us) server_negotiation.cc:407] Connection header received
0620 06:58:17.417051 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.417057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.417144 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.417251 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.418710 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.419868 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.421288 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.421952 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.422082 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.424110 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:17.424117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:17.424122 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:17.424164 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:17.427371 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.428447 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.428453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.428457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.428600 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.428942 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.428945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.428947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.429298 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:17.429424 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:17.429807 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 06:58:17.430052 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:58:17.430485 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.416100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.2:47977)
0620 06:58:17.416736 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:17.416767 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:17.416917 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:17.417296 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:17.417301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:17.417954 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:17.418518 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.418532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.420027 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.420032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.421155 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.421167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.422134 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.422141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.422326 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.423096 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:17.423120 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:17.423917 (+ 797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:17.427588 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.427594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.427597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.428102 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.428714 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.428719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.428722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.428853 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.429408 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:17.429412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:17.429589 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 06:58:17.429733 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 06:58:17.430055 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":347,"threads_started":1}
I20260620 06:58:17.430619 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8196
I20260620 06:58:17.430809 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:17.431394 388 external_mini_cluster.cc:1428] Setting key bcdb51b00b873c70e6dbcf03732c28d6
I20260620 06:58:17.431376 7985 ts_manager.cc:194] Registered new tserver with Master: c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361)
I20260620 06:58:17.432447 7985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58293
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 06:58:17.474759 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34305
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:17.784329 8335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:17.784610 8335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:17.784734 8335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:17.791951 8335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:17.792061 8335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:17.792162 8335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:58:17.792261 8335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:17.801924 8335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:34305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:17.803128 8335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:17.804345 8335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:17.816664 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:17.816689 8343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:17.816629 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:17.817705 8335 server_base.cc:1034] running on GCE node
I20260620 06:58:17.818325 8335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:17.819288 8335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:17.820539 8335 hybrid_clock.cc:648] HybridClock initialized: now 1781938697820470 us; error 35 us; skew 500 ppm
I20260620 06:58:17.820827 8335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:17.824079 8335 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 06:58:17.825479 8335 webserver.cc:460] Webserver started at http://127.0.97.3:36011/ using document root <none> and password file <none>
I20260620 06:58:17.826179 8335 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:17.826277 8335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:17.826633 8335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:17.829535 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "1b07f92a94294b34a8ebb01b42132ab3"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "ff09ecf0dc44d3e804d8f2abd4fc5453"
server_key_iv: "1853d6ff3f8f26d5a278434a1b7b41a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.830305 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1b07f92a94294b34a8ebb01b42132ab3"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "ff09ecf0dc44d3e804d8f2abd4fc5453"
server_key_iv: "1853d6ff3f8f26d5a278434a1b7b41a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.835371 8335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 06:58:17.838706 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:17.840030 8335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:58:17.840175 8335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "1b07f92a94294b34a8ebb01b42132ab3"
format_stamp: "Formatted at 2026-06-20 06:58:17 on dist-test-slave-mz20"
server_key: "ff09ecf0dc44d3e804d8f2abd4fc5453"
server_key_iv: "1853d6ff3f8f26d5a278434a1b7b41a1"
server_key_version: "encryptionkey@0"
I20260620 06:58:17.840372 8335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:17.878574 8335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:17.883018 8335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:17.883298 8335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:17.886442 8335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:17.886593 8335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 06:58:17.886730 8335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:17.886799 8335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:17.908368 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.887154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59207 (local address 127.0.97.62:34305)
0620 06:58:17.887360 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.887367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.888998 (+ 1631us) server_negotiation.cc:407] Connection header received
0620 06:58:17.891201 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.891207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.891295 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.891389 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.893172 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.894175 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.896153 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.896861 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.897034 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.901255 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:17.901263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:17.901269 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:17.901315 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:17.904637 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.905554 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.905560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.905564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.905725 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.906269 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.906275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.906278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.906642 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:17.906787 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:17.907813 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0620 06:58:17.908034 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:58:17.909421 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.887948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.3:59207)
0620 06:58:17.888804 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:17.888844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:17.890897 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:17.891791 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:17.891799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:17.892331 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:17.892983 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.893009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.894445 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.894452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.895977 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.895998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.897140 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.897145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.897337 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.898120 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:17.898145 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:17.900951 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:17.904780 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.904788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.904804 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.905389 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.905868 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.905872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.905876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.906108 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.906863 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:17.906874 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:17.907382 (+ 508us) client_negotiation.cc:769] Sending connection context
0620 06:58:17.907694 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 06:58:17.908255 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":340,"threads_started":1}
I20260620 06:58:17.931360 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.917858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.97.3:40761)
0620 06:58:17.918638 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:17.918661 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:17.918822 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:17.919210 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:17.919215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:17.919850 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:17.920323 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.920341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.921660 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.921665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.922812 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:17.922827 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.923913 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.923923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:17.924066 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.924733 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:17.924753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:17.925464 (+ 711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:17.928647 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.928652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.928656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.929054 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.929413 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:17.929418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:17.929422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:17.929544 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:17.930061 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:17.930066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:17.930301 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 06:58:17.930473 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 06:58:17.930888 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":428,"threads_started":1}
I20260620 06:58:17.931581 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.917805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40761 (local address 127.0.97.62:34305)
0620 06:58:17.917968 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.917974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.918764 (+ 790us) server_negotiation.cc:407] Connection header received
0620 06:58:17.918945 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.918951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.919024 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.919165 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.920486 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.921537 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.922939 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.923548 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.923666 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.925632 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:17.925638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:17.925642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:17.925672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:17.928527 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.929159 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.929165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.929169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.929315 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:17.929637 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:17.929642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:17.929646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:17.929939 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:17.930118 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:17.930972 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 06:58:17.931236 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:58:17.938575 8335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42309
I20260620 06:58:17.938635 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42309 every 8 connection(s)
I20260620 06:58:17.939819 8335 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:17.941783 8470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34305
I20260620 06:58:17.942004 8470 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:17.942734 8470 heartbeater.cc:511] Master 127.0.97.62:34305 requested a full tablet report, sending...
I20260620 06:58:17.944147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8335
I20260620 06:58:17.944217 7986 ts_manager.cc:194] Registered new tserver with Master: 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:17.944330 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:17.944926 388 external_mini_cluster.cc:1428] Setting key d523c6daf66ef9c22ef2d881fed67e79
I20260620 06:58:17.945389 7986 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59207
I20260620 06:58:17.947574 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:58:17.950651 8192 heartbeater.cc:503] Master 127.0.97.62:34305 was elected leader, sending a full tablet report...
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:58:17.993043 8034 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:17.983288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.0.97.62:34305)
0620 06:58:17.983519 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.983525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.983698 (+ 173us) server_negotiation.cc:407] Connection header received
0620 06:58:17.983861 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.983866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.983930 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.984005 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.985260 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.985997 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.987264 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.987719 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.987830 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:17.992771 (+ 4941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54564: BlockingRecv error: recv got EOF from 127.0.0.1:54564 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 06:58:17 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:18.008074 8034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:17.996030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.0.97.62:34305)
0620 06:58:17.996249 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:17.996254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:17.996398 (+ 144us) server_negotiation.cc:407] Connection header received
0620 06:58:17.996527 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:17.996531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:17.996600 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:17.996678 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:17.997685 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.998393 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:17.999526 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:17.999909 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.000024 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.003370 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:18.003375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:18.003378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:18.003412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:18.005720 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:18.006466 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:18.006470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:18.006472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:18.006634 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:18.006954 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:18.006957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:18.006959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:18.007242 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:18.007412 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.007641 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.007802 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 06:58:18.011742 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:18.018810 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:18.042848 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.030808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39045 (local address 127.0.0.1:40622)
0620 06:58:18.033353 (+ 2545us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.033368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.033475 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.035157 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.035161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.035183 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.035569 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.035579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.037477 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.037482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.040047 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.040061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.041943 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.041949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.042136 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.042191 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.042299 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.042479 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"thread_start_us":1042,"threads_started":1}
I20260620 06:58:18.044224 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.030697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41361 (local address 127.0.0.1:46882)
0620 06:58:18.032292 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.032336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.032535 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.034885 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.034893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.034923 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.035413 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.035425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.038514 (+ 3089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.038520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.040574 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.040587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.043422 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.043428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.043603 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.043634 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.043687 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.043865 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":2430,"threads_started":1}
I20260620 06:58:18.044597 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.031347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.0.97.1:39045)
0620 06:58:18.033930 (+ 2583us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.034418 (+ 488us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.034451 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:58:18.034636 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.034645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.034890 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.035740 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.035912 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.037330 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.040288 (+ 2958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.041728 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.043643 (+ 1915us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.043853 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.043938 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.044099 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2442,"thread_start_us":2302,"threads_started":1}
I20260620 06:58:18.045248 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.030836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.0.97.2:41361)
0620 06:58:18.033054 (+ 2218us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.033065 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.033095 (+ 30us) server_negotiation.cc:407] Connection header received
0620 06:58:18.033196 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.033204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.033477 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.035633 (+ 2156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.035845 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.038359 (+ 2514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.040768 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.043202 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.044451 (+ 1249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.044637 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.044734 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.044884 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2061,"thread_start_us":1947,"threads_started":1}
I20260620 06:58:18.048576 8261 tablet_service.cc:1467] Processing CreateTablet for tablet 04667fff91354a1bba508bdeaeb1f27b (DEFAULT_TABLE table=test-table [id=81d9f206ec7a4022976d436912261abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.048566 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 04667fff91354a1bba508bdeaeb1f27b (DEFAULT_TABLE table=test-table [id=81d9f206ec7a4022976d436912261abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.049993 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04667fff91354a1bba508bdeaeb1f27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.049999 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04667fff91354a1bba508bdeaeb1f27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.051323 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.034035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.0.1:57466)
0620 06:58:18.043381 (+ 9346us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.043402 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.043488 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.046301 (+ 2813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.046306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.046329 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.046478 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.046488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.048226 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.048230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.049414 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.049425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.050630 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.050636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.050807 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.050839 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.050924 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.051062 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9240,"thread_start_us":11015,"threads_started":1}
I20260620 06:58:18.051406 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.034389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.0.97.3:42309)
0620 06:58:18.045710 (+ 11321us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.045718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.045750 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:58:18.045836 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.045843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.046101 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.046666 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.046979 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.048105 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.049549 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.050479 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.050616 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.050827 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.050970 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.051109 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11148,"thread_start_us":11023,"threads_started":1}
I20260620 06:58:18.056193 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 04667fff91354a1bba508bdeaeb1f27b (DEFAULT_TABLE table=test-table [id=81d9f206ec7a4022976d436912261abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.057868 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04667fff91354a1bba508bdeaeb1f27b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.066124 8495 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Bootstrap starting.
I20260620 06:58:18.070730 8495 tablet_bootstrap.cc:654] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.071262 8494 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Bootstrap starting.
I20260620 06:58:18.072103 8495 log.cc:824] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:18.072618 8496 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Bootstrap starting.
I20260620 06:58:18.075203 8494 tablet_bootstrap.cc:654] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.075827 8496 tablet_bootstrap.cc:654] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.076262 8495 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: No bootstrap required, opened a new log
I20260620 06:58:18.076257 8494 log.cc:824] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:18.076567 8495 ts_tablet_manager.cc:1397] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260620 06:58:18.077054 8496 log.cc:824] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:18.079073 8494 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: No bootstrap required, opened a new log
I20260620 06:58:18.079638 8494 ts_tablet_manager.cc:1397] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 06:58:18.080150 8496 tablet_bootstrap.cc:492] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: No bootstrap required, opened a new log
I20260620 06:58:18.080417 8496 ts_tablet_manager.cc:1397] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 06:58:18.087007 8495 raft_consensus.cc:348] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.087494 8495 raft_consensus.cc:374] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.087625 8495 raft_consensus.cc:729] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.087401 8494 raft_consensus.cc:348] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.087785 8494 raft_consensus.cc:374] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.087882 8494 raft_consensus.cc:729] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.088384 8494 consensus_queue.cc:260] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.088346 8495 consensus_queue.cc:260] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.089844 8331 heartbeater.cc:503] Master 127.0.97.62:34305 was elected leader, sending a full tablet report...
I20260620 06:58:18.090112 8494 ts_tablet_manager.cc:1428] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 06:58:18.090485 8495 ts_tablet_manager.cc:1428] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260620 06:58:18.090296 8496 raft_consensus.cc:348] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.090737 8496 raft_consensus.cc:374] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.090824 8496 raft_consensus.cc:729] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.091432 8496 consensus_queue.cc:260] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.092969 8470 heartbeater.cc:503] Master 127.0.97.62:34305 was elected leader, sending a full tablet report...
I20260620 06:58:18.093652 8496 ts_tablet_manager.cc:1428] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260620 06:58:18.178817 8332 tablet.cc:2369] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:58:18.191282 8471 tablet.cc:2369] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:58:18.193560 8193 tablet.cc:2369] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:58:18.343320 8500 raft_consensus.cc:482] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:18.343667 8500 raft_consensus.cc:504] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.345324 8500 leader_election.cc:288] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:18.352560 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.346388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38267 (local address 127.0.97.3:42309)
0620 06:58:18.346562 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.346569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.346988 (+ 419us) server_negotiation.cc:407] Connection header received
0620 06:58:18.347159 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.347165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.347241 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.347333 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.347892 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.348737 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.350465 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.351624 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.351750 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.351850 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.352198 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.352322 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:58:18.352639 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.346302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.97.1:38267)
0620 06:58:18.346884 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.346899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.347026 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.347385 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.347390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.347412 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.347744 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.347755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.348858 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.348862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.350319 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.350331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.351827 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.351834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.352010 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.352052 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.352158 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.352292 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":1,"thread_start_us":291,"threads_started":1}
I20260620 06:58:18.352756 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.345762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40001 (local address 127.0.97.2:41361)
0620 06:58:18.345929 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.345933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.346323 (+ 390us) server_negotiation.cc:407] Connection header received
0620 06:58:18.346670 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.346677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.346757 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.346861 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.347417 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.348350 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.350752 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.351867 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.351987 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.352091 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.352405 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.352524 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:58:18.353300 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.345608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41361 (local address 127.0.97.1:40001)
0620 06:58:18.346228 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.346247 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.346440 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.346887 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.346891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.346912 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.347293 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.347306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.348465 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.348471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.350605 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.350619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.352068 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.352074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.352238 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.352270 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.352906 (+ 636us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.353024 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":280,"threads_started":1}
I20260620 06:58:18.353639 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04667fff91354a1bba508bdeaeb1f27b" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3" is_pre_election: true
I20260620 06:58:18.354087 8425 raft_consensus.cc:2454] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:18.354698 8075 leader_election.cc:302] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460; no voters:
I20260620 06:58:18.355064 8500 raft_consensus.cc:2781] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:18.355173 8500 raft_consensus.cc:482] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:18.355265 8500 raft_consensus.cc:3037] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.355146 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04667fff91354a1bba508bdeaeb1f27b" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" is_pre_election: true
I20260620 06:58:18.355584 8281 raft_consensus.cc:2454] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:18.357513 8500 raft_consensus.cc:504] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.358062 8500 leader_election.cc:288] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Requested vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:18.358407 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04667fff91354a1bba508bdeaeb1f27b" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
I20260620 06:58:18.358513 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04667fff91354a1bba508bdeaeb1f27b" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3"
I20260620 06:58:18.358589 8281 raft_consensus.cc:3037] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.358726 8425 raft_consensus.cc:3037] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.360747 8425 raft_consensus.cc:2454] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:18.360803 8281 raft_consensus.cc:2454] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:18.361054 8075 leader_election.cc:302] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460; no voters:
I20260620 06:58:18.361318 8500 raft_consensus.cc:2781] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:18.361709 8500 raft_consensus.cc:686] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Becoming Leader. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Running, Role: LEADER
I20260620 06:58:18.362150 8500 consensus_queue.cc:237] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.366674 7986 catalog_manager.cc:5495] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 reported cstate change: term changed from 0 to 1, leader changed from <none> to c168093cb7d5487eacc71ee1d8124460 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c168093cb7d5487eacc71ee1d8124460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } health_report { overall_health: UNKNOWN } } }
I20260620 06:58:18.405295 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.400574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.97.1:39045)
0620 06:58:18.400748 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.400753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.400822 (+ 69us) server_negotiation.cc:407] Connection header received
0620 06:58:18.401000 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.401004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.401098 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.401238 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:18.401498 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.402190 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.403472 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.403849 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.403940 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.404329 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.404860 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.404978 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.405021 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.405086 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:58:18.412942 8425 raft_consensus.cc:1261] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.412938 8281 raft_consensus.cc:1261] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.413739 8500 consensus_queue.cc:1035] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.414012 8505 consensus_queue.cc:1035] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.424201 8508 mvcc.cc:205] Tried to move back new op lower bound from 7298820908690309120 to 7298820908494684160. Current Snapshot: MvccSnapshot[applied={T|T < 7298820908690309120}]
I20260620 06:58:18.461644 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:18.461915 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:18.465385 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Sending DeleteTablet for 3 replicas of tablet 04667fff91354a1bba508bdeaeb1f27b
I20260620 06:58:18.466485 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 04667fff91354a1bba508bdeaeb1f27b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40622
I20260620 06:58:18.466460 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 04667fff91354a1bba508bdeaeb1f27b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260620 06:58:18.467741 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet 04667fff91354a1bba508bdeaeb1f27b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46882
I20260620 06:58:18.467767 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:18.468402 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:18.469823 8519 tablet_replica.cc:330] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: stopping tablet replica
I20260620 06:58:18.470237 8519 raft_consensus.cc:2229] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:18.470734 8519 raft_consensus.cc:2258] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:18.471395 8517 tablet_replica.cc:330] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: stopping tablet replica
I20260620 06:58:18.471765 8517 raft_consensus.cc:2229] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:18.472143 8517 raft_consensus.cc:2258] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:18.472837 8518 tablet_replica.cc:330] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: stopping tablet replica
I20260620 06:58:18.473186 8518 raft_consensus.cc:2229] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:18.473757 8518 raft_consensus.cc:2258] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:18.475864 8517 ts_tablet_manager.cc:1905] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:18.476085 8518 ts_tablet_manager.cc:1905] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:18.478201 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3602b29fb549898025fda91a23901f (DEFAULT_TABLE table=test-table [id=8c34831112d54b6cb649f544ae86d002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.478664 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3602b29fb549898025fda91a23901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.482937 8517 ts_tablet_manager.cc:1918] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:18.483100 8517 log.cc:1196] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/04667fff91354a1bba508bdeaeb1f27b
I20260620 06:58:18.483302 8518 ts_tablet_manager.cc:1918] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:18.483537 8518 log.cc:1196] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/04667fff91354a1bba508bdeaeb1f27b
I20260620 06:58:18.483584 8517 ts_tablet_manager.cc:1939] T 04667fff91354a1bba508bdeaeb1f27b P 1b07f92a94294b34a8ebb01b42132ab3: Deleting consensus metadata
I20260620 06:58:18.484091 8518 ts_tablet_manager.cc:1939] T 04667fff91354a1bba508bdeaeb1f27b P c168093cb7d5487eacc71ee1d8124460: Deleting consensus metadata
I20260620 06:58:18.484066 8261 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3602b29fb549898025fda91a23901f (DEFAULT_TABLE table=test-table [id=8c34831112d54b6cb649f544ae86d002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.484513 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3602b29fb549898025fda91a23901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.486567 8519 ts_tablet_manager.cc:1905] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:18.486886 7970 catalog_manager.cc:4841] TS c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045): tablet 04667fff91354a1bba508bdeaeb1f27b (table test-table [id=81d9f206ec7a4022976d436912261abe]) successfully deleted
I20260620 06:58:18.487317 7972 catalog_manager.cc:4841] TS 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309): tablet 04667fff91354a1bba508bdeaeb1f27b (table test-table [id=81d9f206ec7a4022976d436912261abe]) successfully deleted
I20260620 06:58:18.488030 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3602b29fb549898025fda91a23901f (DEFAULT_TABLE table=test-table [id=8c34831112d54b6cb649f544ae86d002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:18.488528 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3602b29fb549898025fda91a23901f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.492880 8495 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Bootstrap starting.
I20260620 06:58:18.492903 8496 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Bootstrap starting.
I20260620 06:58:18.493646 8519 ts_tablet_manager.cc:1918] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:18.493837 8519 log.cc:1196] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/04667fff91354a1bba508bdeaeb1f27b
I20260620 06:58:18.494403 8519 ts_tablet_manager.cc:1939] T 04667fff91354a1bba508bdeaeb1f27b P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting consensus metadata
I20260620 06:58:18.494503 8496 tablet_bootstrap.cc:654] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.496295 7972 catalog_manager.cc:4841] TS c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361): tablet 04667fff91354a1bba508bdeaeb1f27b (table test-table [id=81d9f206ec7a4022976d436912261abe]) successfully deleted
I20260620 06:58:18.494946 8495 tablet_bootstrap.cc:654] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.497798 8494 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Bootstrap starting.
I20260620 06:58:18.499583 8494 tablet_bootstrap.cc:654] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.509797 8496 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: No bootstrap required, opened a new log
I20260620 06:58:18.509892 8495 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: No bootstrap required, opened a new log
I20260620 06:58:18.509985 8496 ts_tablet_manager.cc:1397] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260620 06:58:18.510039 8495 ts_tablet_manager.cc:1397] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20260620 06:58:18.510816 8496 raft_consensus.cc:348] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.510850 8495 raft_consensus.cc:348] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.511053 8495 raft_consensus.cc:374] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.511054 8496 raft_consensus.cc:374] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.511138 8495 raft_consensus.cc:729] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.511144 8496 raft_consensus.cc:729] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.511242 8494 tablet_bootstrap.cc:492] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: No bootstrap required, opened a new log
I20260620 06:58:18.511415 8494 ts_tablet_manager.cc:1397] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260620 06:58:18.511426 8495 consensus_queue.cc:260] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.511437 8496 consensus_queue.cc:260] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.512260 8494 raft_consensus.cc:348] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.512516 8494 raft_consensus.cc:374] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.512652 8494 raft_consensus.cc:729] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.512725 8495 ts_tablet_manager.cc:1428] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:18.512987 8494 consensus_queue.cc:260] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.513607 8494 ts_tablet_manager.cc:1428] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:58:18.513636 8496 ts_tablet_manager.cc:1428] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:58:18.672394 8505 raft_consensus.cc:482] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:18.672657 8505 raft_consensus.cc:504] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.673322 8505 leader_election.cc:288] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:18.673949 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3602b29fb549898025fda91a23901f" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" is_pre_election: true
I20260620 06:58:18.674010 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3602b29fb549898025fda91a23901f" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3" is_pre_election: true
I20260620 06:58:18.674211 8281 raft_consensus.cc:2454] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:18.674247 8425 raft_consensus.cc:2454] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:18.674754 8075 leader_election.cc:302] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c168093cb7d5487eacc71ee1d8124460, c92109f5b40b487cbeaef9fa7ae8c6b7; no voters:
I20260620 06:58:18.675062 8505 raft_consensus.cc:2781] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:18.675181 8505 raft_consensus.cc:482] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:18.675287 8505 raft_consensus.cc:3037] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.678400 8505 raft_consensus.cc:504] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.679103 8505 leader_election.cc:288] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Requested vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:18.679582 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3602b29fb549898025fda91a23901f" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3"
I20260620 06:58:18.679575 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3602b29fb549898025fda91a23901f" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
I20260620 06:58:18.679765 8425 raft_consensus.cc:3037] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.679769 8281 raft_consensus.cc:3037] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.681741 8425 raft_consensus.cc:2454] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:18.682119 8075 leader_election.cc:302] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460; no voters:
I20260620 06:58:18.682363 8505 raft_consensus.cc:2781] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:18.682472 8281 raft_consensus.cc:2454] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:18.682521 8505 raft_consensus.cc:686] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Becoming Leader. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Running, Role: LEADER
I20260620 06:58:18.682773 8505 consensus_queue.cc:237] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.685298 7984 catalog_manager.cc:5495] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 reported cstate change: term changed from 0 to 1, leader changed from <none> to c168093cb7d5487eacc71ee1d8124460 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c168093cb7d5487eacc71ee1d8124460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:58:18 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:18.766001 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.752914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.97.62:34305)
0620 06:58:18.753855 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.753860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.753877 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:58:18.753913 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.753916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.753966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.754047 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:18.755261 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.756047 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.757242 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.757626 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.757729 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.761188 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:18.761192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:18.761196 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:18.761223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:18.763612 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:18.764204 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:18.764209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:18.764213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:18.764323 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:18.764635 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:18.764640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:18.764644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:18.764882 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:18.765002 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.765383 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.765658 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":826,"thread_start_us":636,"threads_started":1}
I20260620 06:58:18.766021 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.752851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34305 (local address 127.0.0.1:34548)
0620 06:58:18.753532 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.753562 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.753744 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.754095 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.754099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.754851 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:18.755082 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.755092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.756172 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.756177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.757107 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.757117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.757779 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.757785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.757961 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.758711 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:18.758731 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:18.761001 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:18.763729 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:18.763741 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:18.763750 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:18.764098 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:18.764414 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:18.764419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:18.764422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:18.764545 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:18.764997 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:18.765004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:18.765144 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.765293 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.765538 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":372,"threads_started":1}
I20260620 06:58:18.771148 7984 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:58:18.772133 7984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:18.778712 8122 tablet_service.cc:1467] Processing CreateTablet for tablet eb855ab4056045dfa7b56a3b2e735247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf292beace734d028ff093e4f907d3af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:58:18.779242 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb855ab4056045dfa7b56a3b2e735247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.779119 8400 tablet_service.cc:1467] Processing CreateTablet for tablet eb855ab4056045dfa7b56a3b2e735247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf292beace734d028ff093e4f907d3af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:58:18.779649 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb855ab4056045dfa7b56a3b2e735247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.783481 8261 tablet_service.cc:1467] Processing CreateTablet for tablet eb855ab4056045dfa7b56a3b2e735247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bf292beace734d028ff093e4f907d3af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:58:18.784044 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb855ab4056045dfa7b56a3b2e735247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:18.785758 8494 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: Bootstrap starting.
I20260620 06:58:18.786773 8496 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: Bootstrap starting.
I20260620 06:58:18.787524 8494 tablet_bootstrap.cc:654] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.788365 8496 tablet_bootstrap.cc:654] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.794600 8495 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: Bootstrap starting.
I20260620 06:58:18.796600 8495 tablet_bootstrap.cc:654] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:18.797830 8496 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: No bootstrap required, opened a new log
I20260620 06:58:18.798020 8496 ts_tablet_manager.cc:1397] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260620 06:58:18.798789 8496 raft_consensus.cc:348] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.799053 8496 raft_consensus.cc:374] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.799132 8496 raft_consensus.cc:729] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.799530 8495 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: No bootstrap required, opened a new log
I20260620 06:58:18.799413 8496 consensus_queue.cc:260] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.799676 8495 ts_tablet_manager.cc:1397] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:58:18.799851 8502 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.800120 8494 tablet_bootstrap.cc:492] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: No bootstrap required, opened a new log
I20260620 06:58:18.800210 8502 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.800289 8494 ts_tablet_manager.cc:1397] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260620 06:58:18.800333 8495 raft_consensus.cc:348] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.800547 8495 raft_consensus.cc:374] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.800613 8495 raft_consensus.cc:729] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.800824 8495 consensus_queue.cc:260] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.801007 8494 raft_consensus.cc:348] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.801246 8494 raft_consensus.cc:374] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:18.801316 8494 raft_consensus.cc:729] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Initialized, Role: FOLLOWER
I20260620 06:58:18.801529 8494 consensus_queue.cc:260] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.801501 8501 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.801801 8501 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.802109 8494 ts_tablet_manager.cc:1428] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 06:58:18.802985 8505 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.803241 8505 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.803915 8495 ts_tablet_manager.cc:1428] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 06:58:18.804389 8496 ts_tablet_manager.cc:1428] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 06:58:18.822781 8502 raft_consensus.cc:482] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:18.822988 8502 raft_consensus.cc:504] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.824479 8502 leader_election.cc:288] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045), c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361)
I20260620 06:58:18.832518 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.824846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37001 (local address 127.0.97.2:41361)
0620 06:58:18.825022 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.825028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.826068 (+ 1040us) server_negotiation.cc:407] Connection header received
0620 06:58:18.826326 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.826332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.826409 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.826498 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.827709 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.828517 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.830329 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.831549 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.831677 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.831777 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.832109 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.832243 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:58:18.832592 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.824807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41361 (local address 127.0.97.3:37001)
0620 06:58:18.825972 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.825989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.826159 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.826695 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.826699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.826724 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.827546 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.827560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.828637 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.828642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.830190 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.830201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.831740 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.831746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.831919 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.831954 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.832070 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.832225 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":2,"thread_start_us":916,"threads_started":1}
I20260620 06:58:18.833608 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.825227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57841 (local address 127.0.97.1:39045)
0620 06:58:18.825413 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.825418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.825811 (+ 393us) server_negotiation.cc:407] Connection header received
0620 06:58:18.826113 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.826119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.826205 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.826290 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:18.826928 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.827885 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.830420 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.831700 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.831820 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.831907 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.832413 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.832539 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:58:18.833570 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb855ab4056045dfa7b56a3b2e735247" candidate_uuid: "1b07f92a94294b34a8ebb01b42132ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" is_pre_election: true
I20260620 06:58:18.833834 8281 raft_consensus.cc:2454] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b07f92a94294b34a8ebb01b42132ab3 in term 0.
I20260620 06:58:18.834267 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.824734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39045 (local address 127.0.97.3:57841)
0620 06:58:18.825703 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.825723 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.825928 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.826298 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.826303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.826324 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:18.826579 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.826737 (+ 158us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.828035 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.828040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.830221 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.830235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.832034 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.832039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.832234 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.832264 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.833773 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.833898 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":59,"thread_start_us":691,"threads_started":1}
I20260620 06:58:18.834431 8353 leader_election.cc:302] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c92109f5b40b487cbeaef9fa7ae8c6b7; no voters:
I20260620 06:58:18.834815 8502 raft_consensus.cc:2781] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:18.834916 8502 raft_consensus.cc:482] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:18.835023 8502 raft_consensus.cc:3037] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.835270 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb855ab4056045dfa7b56a3b2e735247" candidate_uuid: "1b07f92a94294b34a8ebb01b42132ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168093cb7d5487eacc71ee1d8124460" is_pre_election: true
I20260620 06:58:18.835713 8142 raft_consensus.cc:2454] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b07f92a94294b34a8ebb01b42132ab3 in term 0.
I20260620 06:58:18.837684 8502 raft_consensus.cc:504] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.838272 8502 leader_election.cc:288] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [CANDIDATE]: Term 1 election: Requested vote from peers c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045), c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361)
I20260620 06:58:18.838699 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb855ab4056045dfa7b56a3b2e735247" candidate_uuid: "1b07f92a94294b34a8ebb01b42132ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168093cb7d5487eacc71ee1d8124460"
I20260620 06:58:18.838735 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb855ab4056045dfa7b56a3b2e735247" candidate_uuid: "1b07f92a94294b34a8ebb01b42132ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
I20260620 06:58:18.838869 8142 raft_consensus.cc:3037] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.838985 8281 raft_consensus.cc:3037] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:18.841230 8142 raft_consensus.cc:2454] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b07f92a94294b34a8ebb01b42132ab3 in term 1.
I20260620 06:58:18.841579 8351 leader_election.cc:302] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460; no voters:
I20260620 06:58:18.841797 8281 raft_consensus.cc:2454] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b07f92a94294b34a8ebb01b42132ab3 in term 1.
I20260620 06:58:18.841847 8502 raft_consensus.cc:2781] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:18.842394 8502 raft_consensus.cc:686] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Running, Role: LEADER
I20260620 06:58:18.842769 8502 consensus_queue.cc:237] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:18.846192 7985 catalog_manager.cc:5495] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } health_report { overall_health: HEALTHY } } }
I20260620 06:58:18.846648 8536 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: TxnStatusTablet state changed. Reason: New leader 1b07f92a94294b34a8ebb01b42132ab3. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.846897 8536 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:58:18.847467 8538 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:58:18.847678 8538 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:58:18.869766 8529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.859311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.0.1:49142)
0620 06:58:18.859656 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.859677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.859813 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.863461 (+ 3648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.863466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.863481 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:58:18.863674 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.863687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.865140 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.865146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.866723 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.866741 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.867817 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.867823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.868029 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.868114 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.869160 (+ 1046us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.869183 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.869266 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.869417 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193}
I20260620 06:58:18.869761 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.859325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.0.97.3:42309)
0620 06:58:18.863001 (+ 3676us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.863007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.863027 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:58:18.863127 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.863132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.863240 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.863407 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:18.863849 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.864989 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.866874 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.867658 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.867784 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.868312 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.869042 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.869199 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.869302 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.869393 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3525,"thread_start_us":3386,"threads_started":1}
I20260620 06:58:18.874482 8281 raft_consensus.cc:1261] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Refusing update from remote peer 1b07f92a94294b34a8ebb01b42132ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.874895 8142 raft_consensus.cc:1261] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Refusing update from remote peer 1b07f92a94294b34a8ebb01b42132ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.875375 8502 consensus_queue.cc:1035] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.875753 8536 consensus_queue.cc:1035] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.889938 8501 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: TxnStatusTablet state changed. Reason: New leader 1b07f92a94294b34a8ebb01b42132ab3. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.890252 8501 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.894212 8502 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.894511 8502 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:58:18.901211 8505 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: TxnStatusTablet state changed. Reason: New leader 1b07f92a94294b34a8ebb01b42132ab3. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.901463 8505 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.903801 8501 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.903832 8502 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.904052 8501 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c92109f5b40b487cbeaef9fa7ae8c6b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.904073 8502 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P 1b07f92a94294b34a8ebb01b42132ab3: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:58:18.905776 8505 tablet_replica.cc:439] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b07f92a94294b34a8ebb01b42132ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } } }
I20260620 06:58:18.906011 8505 tablet_replica.cc:442] T eb855ab4056045dfa7b56a3b2e735247 P c168093cb7d5487eacc71ee1d8124460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:58:18.919224 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.913504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.97.1:35899)
0620 06:58:18.914023 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.914037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.914187 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.914517 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.914520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.914530 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:58:18.914896 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.914905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.916323 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.916327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.917320 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.917331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.918221 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.918225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.918335 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.918385 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.918794 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.918801 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.918856 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.918949 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":256,"threads_started":1}
I20260620 06:58:18.919255 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.913484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35899 (local address 127.0.97.3:42309)
0620 06:58:18.913618 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.913624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.914121 (+ 497us) server_negotiation.cc:407] Connection header received
0620 06:58:18.914315 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.914320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.914400 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.914498 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:18.915052 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.916191 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.917447 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.918085 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.918216 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.918582 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.918709 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.918798 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.918879 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.918952 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:58:18.928386 8425 raft_consensus.cc:1261] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.928398 8281 raft_consensus.cc:1261] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:18.928925 8505 consensus_queue.cc:1035] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.929222 8500 consensus_queue.cc:1035] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:18.937762 8509 mvcc.cc:205] Tried to move back new op lower bound from 7298820910805897216 to 7298820909806858240. Current Snapshot: MvccSnapshot[applied={T|T < 7298820910805897216}]
I20260620 06:58:18.947436 8510 mvcc.cc:205] Tried to move back new op lower bound from 7298820910805897216 to 7298820909806858240. Current Snapshot: MvccSnapshot[applied={T|T < 7298820910805897216}]
I20260620 06:58:18.999974 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.981941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39417 (local address 127.0.97.1:39045)
0620 06:58:18.982112 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:18.982118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:18.994268 (+ 12150us) server_negotiation.cc:407] Connection header received
0620 06:58:18.994527 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:18.994533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:18.994652 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:18.994709 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:18.995504 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.996601 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.998355 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.998817 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.998926 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.999334 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.999431 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.999512 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:18.999632 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 06:58:18.999709 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:58:19.000120 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:18.981504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39045 (local address 127.0.97.3:39417)
0620 06:58:18.994155 (+ 12651us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:18.994178 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:18.994388 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:18.994930 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:18.994934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:18.994947 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:58:18.995346 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.995356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.996762 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.996766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.998217 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:18.998229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:18.998989 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:18.998997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:18.999113 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:18.999179 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:58:18.999518 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:58:18.999529 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 06:58:18.999597 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:58:18.999732 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12171,"thread_start_us":12202,"threads_started":1}
I20260620 06:58:19.030301 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.030570 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.033784 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Sending DeleteTablet for 3 replicas of tablet 4e3602b29fb549898025fda91a23901f
I20260620 06:58:19.034569 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3602b29fb549898025fda91a23901f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40622
I20260620 06:58:19.034735 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3602b29fb549898025fda91a23901f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46882
I20260620 06:58:19.035413 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3602b29fb549898025fda91a23901f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260620 06:58:19.046458 8556 tablet_replica.cc:330] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: stopping tablet replica
I20260620 06:58:19.046797 8556 raft_consensus.cc:2229] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.047200 8556 raft_consensus.cc:2258] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.047972 8554 tablet_replica.cc:330] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: stopping tablet replica
I20260620 06:58:19.048252 8554 raft_consensus.cc:2229] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:19.048602 8554 raft_consensus.cc:2258] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.048717 8556 ts_tablet_manager.cc:1905] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.049513 8554 ts_tablet_manager.cc:1905] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.056031 8554 ts_tablet_manager.cc:1918] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.056259 8554 log.cc:1196] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/4e3602b29fb549898025fda91a23901f
I20260620 06:58:19.056767 8554 ts_tablet_manager.cc:1939] T 4e3602b29fb549898025fda91a23901f P c168093cb7d5487eacc71ee1d8124460: Deleting consensus metadata
I20260620 06:58:19.057981 7970 catalog_manager.cc:4841] TS c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045): tablet 4e3602b29fb549898025fda91a23901f (table test-table [id=8c34831112d54b6cb649f544ae86d002]) successfully deleted
I20260620 06:58:19.060001 8556 ts_tablet_manager.cc:1918] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.060498 8555 tablet_replica.cc:330] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: stopping tablet replica
I20260620 06:58:19.060763 8555 raft_consensus.cc:2229] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.061000 8555 raft_consensus.cc:2258] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.061235 8556 log.cc:1196] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/4e3602b29fb549898025fda91a23901f
I20260620 06:58:19.061705 8556 ts_tablet_manager.cc:1939] T 4e3602b29fb549898025fda91a23901f P 1b07f92a94294b34a8ebb01b42132ab3: Deleting consensus metadata
I20260620 06:58:19.062762 7972 catalog_manager.cc:4841] TS 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309): tablet 4e3602b29fb549898025fda91a23901f (table test-table [id=8c34831112d54b6cb649f544ae86d002]) successfully deleted
I20260620 06:58:19.063601 8555 ts_tablet_manager.cc:1905] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.070358 8555 ts_tablet_manager.cc:1918] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.070489 8555 log.cc:1196] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/4e3602b29fb549898025fda91a23901f
I20260620 06:58:19.070995 8555 ts_tablet_manager.cc:1939] T 4e3602b29fb549898025fda91a23901f P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting consensus metadata
I20260620 06:58:19.072057 7972 catalog_manager.cc:4841] TS c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361): tablet 4e3602b29fb549898025fda91a23901f (table test-table [id=8c34831112d54b6cb649f544ae86d002]) successfully deleted
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:58:19.100157 8530 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:19.089903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.0.97.62:34305)
0620 06:58:19.090110 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.090117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.090478 (+ 361us) server_negotiation.cc:407] Connection header received
0620 06:58:19.090672 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.090678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.090757 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.090873 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:19.092100 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.093221 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.094434 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.095125 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.095266 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.099932 (+ 4666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34552: BlockingRecv error: recv got EOF from 127.0.0.1:34552 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 06:58:19.120309 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.102929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.0.97.62:34305)
0620 06:58:19.103104 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.103110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.103396 (+ 286us) server_negotiation.cc:407] Connection header received
0620 06:58:19.103519 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.103525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.103600 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.103711 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:19.104729 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.105838 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.107092 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.107674 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.107797 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.111635 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:19.111651 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:19.111655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:19.111690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:19.117471 (+ 5781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.118340 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.118345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.118347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.118491 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.119007 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.119011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.119014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.119308 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:19.119495 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.119853 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.120031 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:58:19.124315 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:19.124929 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:19.131315 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 (DEFAULT_TABLE table=test-table [id=7b59c6c291a24ae7bb86fd024a56860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.131584 8261 tablet_service.cc:1467] Processing CreateTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 (DEFAULT_TABLE table=test-table [id=7b59c6c291a24ae7bb86fd024a56860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.131755 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448cce087f74f4f9498929ab4c23fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.131664 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 (DEFAULT_TABLE table=test-table [id=7b59c6c291a24ae7bb86fd024a56860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.131934 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448cce087f74f4f9498929ab4c23fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.132079 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1448cce087f74f4f9498929ab4c23fa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.137933 8495 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Bootstrap starting.
I20260620 06:58:19.137975 8494 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Bootstrap starting.
I20260620 06:58:19.139312 8496 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Bootstrap starting.
I20260620 06:58:19.139490 8494 tablet_bootstrap.cc:654] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.139609 8495 tablet_bootstrap.cc:654] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.140946 8496 tablet_bootstrap.cc:654] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.150902 8495 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: No bootstrap required, opened a new log
I20260620 06:58:19.151085 8496 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: No bootstrap required, opened a new log
I20260620 06:58:19.151124 8495 ts_tablet_manager.cc:1397] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260620 06:58:19.151286 8496 ts_tablet_manager.cc:1397] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260620 06:58:19.151528 8494 tablet_bootstrap.cc:492] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: No bootstrap required, opened a new log
I20260620 06:58:19.151708 8494 ts_tablet_manager.cc:1397] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260620 06:58:19.152101 8495 raft_consensus.cc:348] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.152444 8495 raft_consensus.cc:374] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.152295 8496 raft_consensus.cc:348] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.152551 8496 raft_consensus.cc:374] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.152426 8494 raft_consensus.cc:348] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.152659 8494 raft_consensus.cc:374] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.152661 8496 raft_consensus.cc:729] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.152659 8495 raft_consensus.cc:729] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.152755 8494 raft_consensus.cc:729] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.153008 8494 consensus_queue.cc:260] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.153007 8495 consensus_queue.cc:260] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.153553 8494 ts_tablet_manager.cc:1428] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:58:19.153841 8496 consensus_queue.cc:260] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.154690 8495 ts_tablet_manager.cc:1428] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:19.155587 8496 ts_tablet_manager.cc:1428] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 06:58:19.317801 8505 raft_consensus.cc:482] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:19.318020 8505 raft_consensus.cc:504] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.318691 8505 leader_election.cc:288] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:19.319319 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448cce087f74f4f9498929ab4c23fa2" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3" is_pre_election: true
I20260620 06:58:19.319319 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448cce087f74f4f9498929ab4c23fa2" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" is_pre_election: true
I20260620 06:58:19.319553 8281 raft_consensus.cc:2454] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:19.319552 8425 raft_consensus.cc:2454] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 0.
I20260620 06:58:19.319960 8075 leader_election.cc:302] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c168093cb7d5487eacc71ee1d8124460, c92109f5b40b487cbeaef9fa7ae8c6b7; no voters:
I20260620 06:58:19.320238 8505 raft_consensus.cc:2781] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:19.320356 8505 raft_consensus.cc:482] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:19.320443 8505 raft_consensus.cc:3037] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.322425 8505 raft_consensus.cc:504] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.322962 8505 leader_election.cc:288] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Requested vote from peers c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:19.323393 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448cce087f74f4f9498929ab4c23fa2" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7"
I20260620 06:58:19.323450 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1448cce087f74f4f9498929ab4c23fa2" candidate_uuid: "c168093cb7d5487eacc71ee1d8124460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3"
I20260620 06:58:19.323588 8281 raft_consensus.cc:3037] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.323621 8425 raft_consensus.cc:3037] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.325519 8425 raft_consensus.cc:2454] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:19.325616 8281 raft_consensus.cc:2454] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c168093cb7d5487eacc71ee1d8124460 in term 1.
I20260620 06:58:19.325907 8075 leader_election.cc:302] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460; no voters:
I20260620 06:58:19.326272 8505 raft_consensus.cc:2781] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:19.326421 8505 raft_consensus.cc:686] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Becoming Leader. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Running, Role: LEADER
I20260620 06:58:19.326678 8505 consensus_queue.cc:237] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.329216 7985 catalog_manager.cc:5495] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 reported cstate change: term changed from 0 to 1, leader changed from <none> to c168093cb7d5487eacc71ee1d8124460 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c168093cb7d5487eacc71ee1d8124460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } health_report { overall_health: UNKNOWN } } }
I20260620 06:58:19.348798 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.344155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.97.1:39045)
0620 06:58:19.344348 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.344354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.344432 (+ 78us) server_negotiation.cc:407] Connection header received
0620 06:58:19.344642 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.344647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.344712 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.344784 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:19.345204 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.345914 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.347251 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.347669 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.347771 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.348146 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.348250 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.348320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.348454 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.348525 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:58:19.351078 8425 raft_consensus.cc:1261] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:19.351084 8281 raft_consensus.cc:1261] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Refusing update from remote peer c168093cb7d5487eacc71ee1d8124460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:19.351594 8500 consensus_queue.cc:1035] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:19.351845 8505 consensus_queue.cc:1035] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:19.362226 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.362452 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.365506 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Sending DeleteTablet for 3 replicas of tablet 1448cce087f74f4f9498929ab4c23fa2
I20260620 06:58:19.366236 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40622
I20260620 06:58:19.366313 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46882
I20260620 06:58:19.366336 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1448cce087f74f4f9498929ab4c23fa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260620 06:58:19.366600 8554 tablet_replica.cc:330] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: stopping tablet replica
I20260620 06:58:19.366600 8555 tablet_replica.cc:330] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: stopping tablet replica
I20260620 06:58:19.366693 8556 tablet_replica.cc:330] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: stopping tablet replica
I20260620 06:58:19.366788 8554 raft_consensus.cc:2229] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:19.366783 8555 raft_consensus.cc:2229] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.366870 8556 raft_consensus.cc:2229] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.366979 8555 raft_consensus.cc:2258] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.367056 8556 raft_consensus.cc:2258] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.367054 8554 raft_consensus.cc:2258] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.368024 8555 ts_tablet_manager.cc:1905] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.368023 8556 ts_tablet_manager.cc:1905] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.374311 8555 ts_tablet_manager.cc:1918] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:19.374338 8556 ts_tablet_manager.cc:1918] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:19.374472 8556 log.cc:1196] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/1448cce087f74f4f9498929ab4c23fa2
I20260620 06:58:19.374473 8555 log.cc:1196] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/1448cce087f74f4f9498929ab4c23fa2
I20260620 06:58:19.374883 8556 ts_tablet_manager.cc:1939] T 1448cce087f74f4f9498929ab4c23fa2 P 1b07f92a94294b34a8ebb01b42132ab3: Deleting consensus metadata
I20260620 06:58:19.374914 8555 ts_tablet_manager.cc:1939] T 1448cce087f74f4f9498929ab4c23fa2 P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting consensus metadata
I20260620 06:58:19.376020 7972 catalog_manager.cc:4841] TS c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361): tablet 1448cce087f74f4f9498929ab4c23fa2 (table test-table [id=7b59c6c291a24ae7bb86fd024a56860b]) successfully deleted
I20260620 06:58:19.376324 8554 ts_tablet_manager.cc:1905] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.377488 7986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:19.378140 7986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:19.380686 7972 catalog_manager.cc:4841] TS 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309): tablet 1448cce087f74f4f9498929ab4c23fa2 (table test-table [id=7b59c6c291a24ae7bb86fd024a56860b]) successfully deleted
I20260620 06:58:19.381652 8554 ts_tablet_manager.cc:1918] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:19.381804 8554 log.cc:1196] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/1448cce087f74f4f9498929ab4c23fa2
I20260620 06:58:19.382157 8554 ts_tablet_manager.cc:1939] T 1448cce087f74f4f9498929ab4c23fa2 P c168093cb7d5487eacc71ee1d8124460: Deleting consensus metadata
I20260620 06:58:19.383127 7970 catalog_manager.cc:4841] TS c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045): tablet 1448cce087f74f4f9498929ab4c23fa2 (table test-table [id=7b59c6c291a24ae7bb86fd024a56860b]) successfully deleted
I20260620 06:58:19.385385 8261 tablet_service.cc:1467] Processing CreateTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd (DEFAULT_TABLE table=test-table [id=ae2946c58cc44c668034b6cb9cf84e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.385361 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd (DEFAULT_TABLE table=test-table [id=ae2946c58cc44c668034b6cb9cf84e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.385748 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57477d7c6b4b43bab1c67d52fff844cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.385782 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57477d7c6b4b43bab1c67d52fff844cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.389910 8495 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Bootstrap starting.
I20260620 06:58:19.390288 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd (DEFAULT_TABLE table=test-table [id=ae2946c58cc44c668034b6cb9cf84e42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:19.390720 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57477d7c6b4b43bab1c67d52fff844cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:19.391394 8495 tablet_bootstrap.cc:654] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.391621 8496 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Bootstrap starting.
I20260620 06:58:19.392958 8496 tablet_bootstrap.cc:654] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.397440 8494 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Bootstrap starting.
I20260620 06:58:19.398952 8496 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: No bootstrap required, opened a new log
I20260620 06:58:19.398973 8494 tablet_bootstrap.cc:654] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:19.399092 8496 ts_tablet_manager.cc:1397] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 06:58:19.399986 8496 raft_consensus.cc:348] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.400225 8496 raft_consensus.cc:374] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.400298 8496 raft_consensus.cc:729] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b07f92a94294b34a8ebb01b42132ab3, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.400522 8496 consensus_queue.cc:260] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.401620 8494 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: No bootstrap required, opened a new log
I20260620 06:58:19.401623 8496 ts_tablet_manager.cc:1428] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:58:19.401787 8494 ts_tablet_manager.cc:1397] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 06:58:19.402284 8494 raft_consensus.cc:348] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.402451 8494 raft_consensus.cc:374] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.402494 8494 raft_consensus.cc:729] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c168093cb7d5487eacc71ee1d8124460, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.402730 8494 consensus_queue.cc:260] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.403936 8494 ts_tablet_manager.cc:1428] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:58:19.404600 8495 tablet_bootstrap.cc:492] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: No bootstrap required, opened a new log
I20260620 06:58:19.404776 8495 ts_tablet_manager.cc:1397] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.002s
I20260620 06:58:19.406015 8495 raft_consensus.cc:348] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.406200 8495 raft_consensus.cc:374] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:19.406270 8495 raft_consensus.cc:729] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Initialized, Role: FOLLOWER
I20260620 06:58:19.406652 8495 consensus_queue.cc:260] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.407155 8495 ts_tablet_manager.cc:1428] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:58:19.584190 8501 raft_consensus.cc:482] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:19.584409 8501 raft_consensus.cc:504] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.585984 8501 leader_election.cc:288] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:19.593966 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.586322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43315 (local address 127.0.97.1:39045)
0620 06:58:19.586511 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.586517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.587144 (+ 627us) server_negotiation.cc:407] Connection header received
0620 06:58:19.587433 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.587440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.587517 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.587577 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:19.588401 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.589419 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.591910 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.593060 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.593190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.593296 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.593598 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.593714 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:58:19.594254 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.586483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54077 (local address 127.0.97.3:42309)
0620 06:58:19.587918 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.587922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.587936 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:58:19.587988 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.587991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.588047 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.588761 (+ 714us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:19.588918 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.589781 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.592120 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.593320 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.593415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.593487 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.593833 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.593966 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":1263,"threads_started":1}
I20260620 06:58:19.594328 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.586397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.97.2:54077)
0620 06:58:19.587390 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:19.587408 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:19.587534 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:19.588194 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:19.588199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:19.588231 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:19.588597 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.588608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.589921 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.589926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.591981 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.591990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.593513 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.593519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.593688 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.593715 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:58:19.593793 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:58:19.593915 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":21,"thread_start_us":621,"threads_started":1}
I20260620 06:58:19.594501 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.586198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39045 (local address 127.0.97.2:43315)
0620 06:58:19.587021 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:19.587050 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:19.587264 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:19.587820 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:19.587824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:19.587849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:19.588110 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.588122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.589576 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.589580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.591726 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.591742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.593243 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.593247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.593423 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.593464 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:58:19.593565 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 06:58:19.593708 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":56,"thread_start_us":479,"threads_started":1}
I20260620 06:58:19.595054 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57477d7c6b4b43bab1c67d52fff844cd" candidate_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168093cb7d5487eacc71ee1d8124460" is_pre_election: true
I20260620 06:58:19.595054 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57477d7c6b4b43bab1c67d52fff844cd" candidate_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3" is_pre_election: true
I20260620 06:58:19.595273 8142 raft_consensus.cc:2454] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92109f5b40b487cbeaef9fa7ae8c6b7 in term 0.
I20260620 06:58:19.595274 8425 raft_consensus.cc:2454] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c92109f5b40b487cbeaef9fa7ae8c6b7 in term 0.
I20260620 06:58:19.595752 8212 leader_election.cc:302] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c168093cb7d5487eacc71ee1d8124460, c92109f5b40b487cbeaef9fa7ae8c6b7; no voters:
I20260620 06:58:19.596119 8501 raft_consensus.cc:2781] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:19.596264 8501 raft_consensus.cc:482] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:19.596362 8501 raft_consensus.cc:3037] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.598479 8501 raft_consensus.cc:504] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.599071 8501 leader_election.cc:288] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [CANDIDATE]: Term 1 election: Requested vote from peers c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045), 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309)
I20260620 06:58:19.599530 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57477d7c6b4b43bab1c67d52fff844cd" candidate_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b07f92a94294b34a8ebb01b42132ab3"
I20260620 06:58:19.599535 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57477d7c6b4b43bab1c67d52fff844cd" candidate_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c168093cb7d5487eacc71ee1d8124460"
I20260620 06:58:19.599695 8425 raft_consensus.cc:3037] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.599726 8142 raft_consensus.cc:3037] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:19.601578 8425 raft_consensus.cc:2454] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92109f5b40b487cbeaef9fa7ae8c6b7 in term 1.
I20260620 06:58:19.601743 8142 raft_consensus.cc:2454] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c92109f5b40b487cbeaef9fa7ae8c6b7 in term 1.
I20260620 06:58:19.601986 8214 leader_election.cc:302] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b07f92a94294b34a8ebb01b42132ab3, c92109f5b40b487cbeaef9fa7ae8c6b7; no voters:
I20260620 06:58:19.602319 8501 raft_consensus.cc:2781] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:19.602737 8501 raft_consensus.cc:686] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 LEADER]: Becoming Leader. State: Replica: c92109f5b40b487cbeaef9fa7ae8c6b7, State: Running, Role: LEADER
I20260620 06:58:19.603142 8501 consensus_queue.cc:237] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } }
I20260620 06:58:19.610175 7984 catalog_manager.cc:5495] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c92109f5b40b487cbeaef9fa7ae8c6b7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 } health_report { overall_health: UNKNOWN } } }
I20260620 06:58:19.651960 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.646340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.97.2:41361)
0620 06:58:19.646958 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.646963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.646985 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:58:19.647087 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.647092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.647187 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.647355 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:19.647756 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.648528 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.649802 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.650204 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.650310 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.650808 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.651362 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.651491 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.651575 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.651648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":309,"threads_started":1}
I20260620 06:58:19.666254 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.661253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47681 (local address 127.0.97.3:42309)
0620 06:58:19.661465 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.661472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.661895 (+ 423us) server_negotiation.cc:407] Connection header received
0620 06:58:19.662069 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.662073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.662135 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.662206 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:19.662673 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.663358 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.664683 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.665094 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.665199 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.665598 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.665699 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.665792 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.665900 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.665974 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 06:58:19.666381 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.661208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42309 (local address 127.0.97.2:47681)
0620 06:58:19.661795 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:19.661811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:19.661948 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:19.662237 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:19.662241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:19.662252 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:58:19.662522 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.662530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.663484 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.663489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.664554 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.664566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.665260 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.665267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.665386 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.665439 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.665793 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.665805 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 06:58:19.665874 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:58:19.666003 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260620 06:58:19.674823 8142 raft_consensus.cc:1261] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Refusing update from remote peer c92109f5b40b487cbeaef9fa7ae8c6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:19.674829 8425 raft_consensus.cc:1261] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Refusing update from remote peer c92109f5b40b487cbeaef9fa7ae8c6b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:19.675515 8501 consensus_queue.cc:1035] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b07f92a94294b34a8ebb01b42132ab3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:19.675787 8582 consensus_queue.cc:1035] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c168093cb7d5487eacc71ee1d8124460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:19.711899 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.707026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52063 (local address 127.0.97.2:41361)
0620 06:58:19.707205 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.707211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.707648 (+ 437us) server_negotiation.cc:407] Connection header received
0620 06:58:19.707835 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.707840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.707908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.707977 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:19.708364 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.709041 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.710301 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.710725 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.710829 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.711174 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.711310 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.711382 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.711527 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.711600 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:58:19.712009 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.706958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41361 (local address 127.0.97.3:52063)
0620 06:58:19.707561 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:19.707574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:19.707697 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:19.708006 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:19.708010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:19.708021 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:58:19.708245 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.708253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.709170 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.709175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.710172 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:19.710183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.710859 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.710864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:19.711002 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.711036 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:58:19.711397 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:58:19.711408 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 06:58:19.711479 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:58:19.711613 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":313,"threads_started":1}
I20260620 06:58:19.741464 7986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.741747 7986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:19.744776 7986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92ce1730d47430aa9321f0446d8059d: Sending DeleteTablet for 3 replicas of tablet 57477d7c6b4b43bab1c67d52fff844cd
I20260620 06:58:19.745602 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40622
I20260620 06:58:19.745667 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57466
I20260620 06:58:19.745909 8554 tablet_replica.cc:330] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: stopping tablet replica
I20260620 06:58:19.745884 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet 57477d7c6b4b43bab1c67d52fff844cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:19 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46882
I20260620 06:58:19.745944 8556 tablet_replica.cc:330] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: stopping tablet replica
I20260620 06:58:19.746078 8554 raft_consensus.cc:2229] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.746098 8556 raft_consensus.cc:2229] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:19.746136 8555 tablet_replica.cc:330] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: stopping tablet replica
I20260620 06:58:19.746273 8554 raft_consensus.cc:2258] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.746297 8556 raft_consensus.cc:2258] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.746289 8555 raft_consensus.cc:2229] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:19.746810 8555 raft_consensus.cc:2258] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:19.747794 8556 ts_tablet_manager.cc:1905] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.748488 8554 ts_tablet_manager.cc:1905] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.751437 8555 ts_tablet_manager.cc:1905] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:19.752971 8556 ts_tablet_manager.cc:1918] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.753091 8556 log.cc:1196] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/57477d7c6b4b43bab1c67d52fff844cd
I20260620 06:58:19.753458 8556 ts_tablet_manager.cc:1939] T 57477d7c6b4b43bab1c67d52fff844cd P 1b07f92a94294b34a8ebb01b42132ab3: Deleting consensus metadata
I20260620 06:58:19.754462 7972 catalog_manager.cc:4841] TS 1b07f92a94294b34a8ebb01b42132ab3 (127.0.97.3:42309): tablet 57477d7c6b4b43bab1c67d52fff844cd (table test-table [id=ae2946c58cc44c668034b6cb9cf84e42]) successfully deleted
I20260620 06:58:19.754895 8554 ts_tablet_manager.cc:1918] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.755023 8554 log.cc:1196] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/57477d7c6b4b43bab1c67d52fff844cd
I20260620 06:58:19.755062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8057
I20260620 06:58:19.755460 8554 ts_tablet_manager.cc:1939] T 57477d7c6b4b43bab1c67d52fff844cd P c168093cb7d5487eacc71ee1d8124460: Deleting consensus metadata
I20260620 06:58:19.756544 7970 catalog_manager.cc:4841] TS c168093cb7d5487eacc71ee1d8124460 (127.0.97.1:39045): tablet 57477d7c6b4b43bab1c67d52fff844cd (table test-table [id=ae2946c58cc44c668034b6cb9cf84e42]) successfully deleted
I20260620 06:58:19.760854 8555 ts_tablet_manager.cc:1918] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:58:19.761005 8555 log.cc:1196] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/57477d7c6b4b43bab1c67d52fff844cd
I20260620 06:58:19.761436 8555 ts_tablet_manager.cc:1939] T 57477d7c6b4b43bab1c67d52fff844cd P c92109f5b40b487cbeaef9fa7ae8c6b7: Deleting consensus metadata
I20260620 06:58:19.762612 7972 catalog_manager.cc:4841] TS c92109f5b40b487cbeaef9fa7ae8c6b7 (127.0.97.2:41361): tablet 57477d7c6b4b43bab1c67d52fff844cd (table test-table [id=ae2946c58cc44c668034b6cb9cf84e42]) successfully deleted
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 06:58:19.771112 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.756184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.0.97.1:39045)
0620 06:58:19.756374 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.756380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.758028 (+ 1648us) server_negotiation.cc:407] Connection header received
0620 06:58:19.758188 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.758192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.758249 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.758336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:19.759473 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.760187 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.761439 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.761872 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.761982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.765879 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:19.765892 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:19.765907 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:19.765939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:19.768596 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.769331 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.769338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.769340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.769445 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.769800 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.769803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.769805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.770132 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:19.770376 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.770681 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.770859 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:58:19.771778 8187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:19.892216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8057
I20260620 06:58:19.920442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8196
Jun 20 06:58:19 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 06:58:19.936149 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:19.921154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.0.97.2:41361)
0620 06:58:19.921340 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:19.921346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:19.921368 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:58:19.921523 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:19.921528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:19.921609 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:19.921747 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:19.922990 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.924133 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.925401 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:19.926032 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:19.926158 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:19.930423 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:19.930434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:19.930448 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:19.930480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:19.933490 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.934264 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.934271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.934273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.934431 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:19.934826 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:19.934830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:19.934833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:19.935197 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:19.935446 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:19.935725 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 06:58:19.935884 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:58:19.936764 8326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:20.058192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8196
I20260620 06:58:20.074566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8335
Jun 20 06:58:20 dist-test-slave-mz20 krb5kdc[7941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 06:58:20.089164 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:20.075349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.97.3:42309)
0620 06:58:20.075523 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:20.075528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:20.075547 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:58:20.075701 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:20.075705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:20.075765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:20.075867 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:20.077104 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:20.077844 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:20.079273 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:20.079703 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:20.079803 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:20.083995 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:20.084010 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:20.084026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:20.084070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:20.086714 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:20.087378 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:20.087386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:20.087388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:20.087522 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:20.087887 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:20.087891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:20.087894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:20.088299 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:20.088516 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:20.088734 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:58:20.088904 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:58:20.089869 8465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:20.209128 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8335
I20260620 06:58:20.227730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 7954
I20260620 06:58:20.228821 8026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:20.354980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 7954
2026-06-20T06:58:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5600 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:20 dist-test-slave-mz20 krb5kdc[8605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:20 dist-test-slave-mz20 krb5kdc[8605](info): set up 2 sockets
Jun 20 06:58:20 dist-test-slave-mz20 krb5kdc[8605](info): commencing operation
krb5kdc: starting...
W20260620 06:58:21.359218 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:21 dist-test-slave-mz20 krb5kdc[8605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:58:21.563452 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42541
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:21.870826 8618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:21.871114 8618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:21.871263 8618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:21.878196 8618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:21.878310 8618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:21.878387 8618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:21.878460 8618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:21.878527 8618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:21.888118 8618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:21.889323 8618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:21.890633 8618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:21.902289 8623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:21.902228 8626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:21.902243 8624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:21.903035 8618 server_base.cc:1034] running on GCE node
I20260620 06:58:21.903759 8618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:21.905081 8618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:21.906276 8618 hybrid_clock.cc:648] HybridClock initialized: now 1781938701906222 us; error 46 us; skew 500 ppm
I20260620 06:58:21.906634 8618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T06:58:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1616 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:22 dist-test-slave-mz20 krb5kdc[8632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:22 dist-test-slave-mz20 krb5kdc[8632](info): set up 2 sockets
Jun 20 06:58:22 dist-test-slave-mz20 krb5kdc[8632](info): commencing operation
krb5kdc: starting...
W20260620 06:58:22.971169 388 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:23 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:58:23.173403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:23.482326 8645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:23.482678 8645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:23.482829 8645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:23.490082 8645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:23.490213 8645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:23.490293 8645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:23.490367 8645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:23.490434 8645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:23.500424 8645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:23.501742 8645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:23.503084 8645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:23.514614 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:23.514659 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:23.514681 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:23.515431 8645 server_base.cc:1034] running on GCE node
I20260620 06:58:23.516207 8645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:23.517721 8645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:23.518925 8645 hybrid_clock.cc:648] HybridClock initialized: now 1781938703518872 us; error 50 us; skew 500 ppm
I20260620 06:58:23.519248 8645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:23 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:23.522588 8645 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:58:23.523986 8645 webserver.cc:460] Webserver started at http://127.0.97.62:42165/ using document root <none> and password file <none>
I20260620 06:58:23.524660 8645 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:23.524770 8645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:23.525082 8645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:23.528040 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "31012f5c86d94e158a1572ec81e7cc36"
format_stamp: "Formatted at 2026-06-20 06:58:23 on dist-test-slave-mz20"
server_key: "ac523c747e7e28d5d9c3313a9a09face"
server_key_iv: "b8526eb247140878bf6db59ce15e2097"
server_key_version: "encryptionkey@0"
I20260620 06:58:23.528782 8645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "31012f5c86d94e158a1572ec81e7cc36"
format_stamp: "Formatted at 2026-06-20 06:58:23 on dist-test-slave-mz20"
server_key: "ac523c747e7e28d5d9c3313a9a09face"
server_key_iv: "b8526eb247140878bf6db59ce15e2097"
server_key_version: "encryptionkey@0"
I20260620 06:58:23.533814 8645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 06:58:23.537468 8660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:23.538934 8645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:23.539079 8645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "31012f5c86d94e158a1572ec81e7cc36"
format_stamp: "Formatted at 2026-06-20 06:58:23 on dist-test-slave-mz20"
server_key: "ac523c747e7e28d5d9c3313a9a09face"
server_key_iv: "b8526eb247140878bf6db59ce15e2097"
server_key_version: "encryptionkey@0"
I20260620 06:58:23.539259 8645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:23.556968 8645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:23.560779 8645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:23.561046 8645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:23.577059 8645 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33079
I20260620 06:58:23.577083 8711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33079 every 8 connection(s)
I20260620 06:58:23.578339 8645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:23.581614 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8645
I20260620 06:58:23.581820 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:23.582038 8712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:23.582469 388 external_mini_cluster.cc:1428] Setting key 8678165e545402fff3e91b10b023d0e4
I20260620 06:58:23.595417 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Bootstrap starting.
Jun 20 06:58:23 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:23.599110 8712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:23.600458 8712 log.cc:824] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:23.603854 8712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: No bootstrap required, opened a new log
I20260620 06:58:23.605402 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:23.584476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.0.97.62:33079)
0620 06:58:23.585236 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:23.585251 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:23.585298 (+ 47us) server_negotiation.cc:407] Connection header received
0620 06:58:23.586880 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:23.586910 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:23.587408 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:23.587855 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:23.589042 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:23.590898 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:23.592117 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:23.593179 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:23.593351 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:23.597164 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:23.597175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:23.597196 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:23.597249 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:23.600843 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:23.601666 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:23.601673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:23.601679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:23.601885 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:23.602227 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:23.602232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:23.602235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:23.603019 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:23.603308 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:23.603672 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 06:58:23.604101 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":321,"threads_started":1}
I20260620 06:58:23.612000 8712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:23.612335 8712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:23.612407 8712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31012f5c86d94e158a1572ec81e7cc36, State: Initialized, Role: FOLLOWER
I20260620 06:58:23.612882 8712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:23.613067 8712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:23.613173 8712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:23.613283 8712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:23.615756 8712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:23.616186 8712 leader_election.cc:302] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31012f5c86d94e158a1572ec81e7cc36; no voters:
I20260620 06:58:23.616655 8712 leader_election.cc:288] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:23.616775 8717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:23.617357 8717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 LEADER]: Becoming Leader. State: Replica: 31012f5c86d94e158a1572ec81e7cc36, State: Running, Role: LEADER
I20260620 06:58:23.617832 8717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:23.618471 8712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:23.620517 8718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31012f5c86d94e158a1572ec81e7cc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260620 06:58:23.620433 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31012f5c86d94e158a1572ec81e7cc36. Latest consensus state: current_term: 1 leader_uuid: "31012f5c86d94e158a1572ec81e7cc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260620 06:58:23.620942 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:23.620962 8718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:23.628010 8723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:23.632272 8723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:23.640146 8723 catalog_manager.cc:1347] Generated new cluster ID: b2c59527dedb415d9c9ab69e0f6bf28a
I20260620 06:58:23.640264 8723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:23.648826 8723 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:23.650076 8723 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:23.657526 8723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Generated new TSK 0
I20260620 06:58:23.658327 8723 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:58:23.709934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:24.025826 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:24.026141 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:24.026258 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:24.033653 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:24.033797 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:24.033965 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:24.044116 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:24.045485 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:24.046869 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:24.059418 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:24.059386 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:24.059448 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:24.060169 8738 server_base.cc:1034] running on GCE node
I20260620 06:58:24.060739 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:24.061587 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:24.062773 8738 hybrid_clock.cc:648] HybridClock initialized: now 1781938704062719 us; error 50 us; skew 500 ppm
I20260620 06:58:24.063064 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:24 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:24.066066 8738 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:58:24.067399 8738 webserver.cc:460] Webserver started at http://127.0.97.1:45015/ using document root <none> and password file <none>
I20260620 06:58:24.068017 8738 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:24.068118 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:24.068403 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:24.071141 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "22fefa68552c40e3a28e7a191ae8f461"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "0cfcd8ba80d9fef5d8cdf2974379b720"
server_key_iv: "b59c35bb8a79bc57b5b6aef16432d99e"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.071863 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "22fefa68552c40e3a28e7a191ae8f461"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "0cfcd8ba80d9fef5d8cdf2974379b720"
server_key_iv: "b59c35bb8a79bc57b5b6aef16432d99e"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.076793 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:58:24.080232 8753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:24.081555 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:58:24.081696 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "22fefa68552c40e3a28e7a191ae8f461"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "0cfcd8ba80d9fef5d8cdf2974379b720"
server_key_iv: "b59c35bb8a79bc57b5b6aef16432d99e"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.081878 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:24.114207 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:24.122220 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:24.122488 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:24.123461 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:24.124722 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:24.124800 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:24.124925 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:24.124974 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:24.153868 8738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39601
I20260620 06:58:24.153904 8865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39601 every 8 connection(s)
I20260620 06:58:24.155139 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:24.162351 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8738
I20260620 06:58:24.162595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:24.163226 388 external_mini_cluster.cc:1428] Setting key 26d6f290aaf3d4dff2e7d8bd69539d0a
Jun 20 06:58:24 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:24.175297 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:24.157592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45531 (local address 127.0.97.62:33079)
0620 06:58:24.157991 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:24.157995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:24.159010 (+ 1015us) server_negotiation.cc:407] Connection header received
0620 06:58:24.160886 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:24.160892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:24.160971 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:24.161067 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:24.162879 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.163611 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.165422 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.165808 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.165908 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:24.169593 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:24.169599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:24.169603 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:24.169641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:24.172380 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:24.173222 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:24.173226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:24.173229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:24.173332 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:24.173726 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:24.173729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:24.173731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:24.173990 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:24.174089 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:24.174897 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 06:58:24.175059 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":207,"threads_started":1}
I20260620 06:58:24.176664 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:24.158177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:45531)
0620 06:58:24.158852 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:24.158894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:24.160610 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:24.161310 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:24.161319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:24.161823 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:24.162649 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:24.162680 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.163760 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.163764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:24.165287 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:24.165300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.166031 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.166038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:24.166313 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:24.167025 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:24.167053 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:24.169378 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:24.172512 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:24.172522 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:24.172541 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:24.173123 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:24.173417 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:24.173421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:24.173424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:24.173639 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:24.174093 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:24.174101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:24.174509 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 06:58:24.174828 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 06:58:24.175331 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
I20260620 06:58:24.178180 8866 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260620 06:58:24.178457 8866 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:24.179159 8866 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260620 06:58:24.181097 8677 ts_manager.cc:194] Registered new tserver with Master: 22fefa68552c40e3a28e7a191ae8f461 (127.0.97.1:39601)
I20260620 06:58:24.182462 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45531
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:58:24.209260 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:24.522667 8872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:24.522964 8872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:24.523097 8872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:24.530391 8872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:24.530517 8872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:24.530694 8872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:24.540896 8872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:24.542107 8872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:24.543409 8872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:24.555671 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:24.555698 8878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:24.555752 8877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:24.556808 8872 server_base.cc:1034] running on GCE node
I20260620 06:58:24.557403 8872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:24.558377 8872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:24.559572 8872 hybrid_clock.cc:648] HybridClock initialized: now 1781938704559530 us; error 65 us; skew 500 ppm
I20260620 06:58:24.559864 8872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:24 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:24.562845 8872 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:58:24.564226 8872 webserver.cc:460] Webserver started at http://127.0.97.2:44515/ using document root <none> and password file <none>
I20260620 06:58:24.564877 8872 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:24.564987 8872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:24.565263 8872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:24.568066 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "2b2a4f3ceb75410ebb8f4e36a74a896d"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "c8776655e9bc2d69a1c5153acc383c35"
server_key_iv: "8512813a16dff0c9fbcaf08bfa809d32"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.568799 8872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2b2a4f3ceb75410ebb8f4e36a74a896d"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "c8776655e9bc2d69a1c5153acc383c35"
server_key_iv: "8512813a16dff0c9fbcaf08bfa809d32"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.573627 8872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:58:24.577049 8887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:24.578433 8872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:24.578595 8872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "2b2a4f3ceb75410ebb8f4e36a74a896d"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "c8776655e9bc2d69a1c5153acc383c35"
server_key_iv: "8512813a16dff0c9fbcaf08bfa809d32"
server_key_version: "encryptionkey@0"
I20260620 06:58:24.578794 8872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:24.596215 8872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:24.599892 8872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:24.600154 8872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:24.601073 8872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:24.602363 8872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:24.602432 8872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:24.602567 8872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:24.602622 8872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 06:58:24.631989 8872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43115
I20260620 06:58:24.632025 8999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43115 every 8 connection(s)
I20260620 06:58:24.633220 8872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:24.639941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 8872
I20260620 06:58:24.640142 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:24.640650 388 external_mini_cluster.cc:1428] Setting key e25d4c7fc39607438bef3f10e612161f
Jun 20 06:58:24 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:24.653795 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:24.635667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59617 (local address 127.0.97.62:33079)
0620 06:58:24.635883 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:24.635890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:24.637081 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 06:58:24.639216 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:24.639221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:24.639290 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:24.639365 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:24.641304 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.642254 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.644389 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.644800 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.644890 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:24.648639 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:24.648644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:24.648647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:24.648676 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:24.651020 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:24.651806 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:24.651809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:24.651811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:24.651924 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:24.652319 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:24.652322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:24.652324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:24.652559 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:24.652670 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:24.653391 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 06:58:24.653561 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:58:24.655206 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:24.636203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:59617)
0620 06:58:24.636918 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:24.636963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:24.638911 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:24.639567 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:24.639579 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:24.640321 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:24.641105 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:24.641138 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.642407 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.642412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:24.644264 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:24.644278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:24.645012 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:24.645018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:24.645283 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:24.645940 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:24.645970 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:24.648412 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:24.651145 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:24.651154 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:24.651171 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:24.651709 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:24.652008 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:24.652012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:24.652016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:24.652232 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:24.652658 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:24.652666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:24.653059 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 06:58:24.653335 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 06:58:24.653845 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260620 06:58:24.656566 9000 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260620 06:58:24.656836 9000 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:24.657519 9000 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260620 06:58:24.658942 8677 ts_manager.cc:194] Registered new tserver with Master: 2b2a4f3ceb75410ebb8f4e36a74a896d (127.0.97.2:43115)
I20260620 06:58:24.659670 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59617
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:58:24.687038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:24.998829 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:24.999114 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:24.999205 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:25.006269 9005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:25.006392 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:25.006511 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:25.016372 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:25.017585 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:25.018981 9005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:25.031390 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:25.031533 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:25.031546 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:25.032541 9005 server_base.cc:1034] running on GCE node
I20260620 06:58:25.033172 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:25.034096 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:25.035281 9005 hybrid_clock.cc:648] HybridClock initialized: now 1781938705035220 us; error 51 us; skew 500 ppm
I20260620 06:58:25.035567 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:25 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:25.038681 9005 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:58:25.040071 9005 webserver.cc:460] Webserver started at http://127.0.97.3:36593/ using document root <none> and password file <none>
I20260620 06:58:25.040709 9005 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:25.040812 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:25.041124 9005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:25.044026 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "a3ea126f86d944ab857ed374c9c639f5"
format_stamp: "Formatted at 2026-06-20 06:58:25 on dist-test-slave-mz20"
server_key: "c772e368c20d69cfcf6e8498b7d00eef"
server_key_iv: "95ccbbf0a3200585f8fa6eb3c92df5de"
server_key_version: "encryptionkey@0"
I20260620 06:58:25.044785 9005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a3ea126f86d944ab857ed374c9c639f5"
format_stamp: "Formatted at 2026-06-20 06:58:25 on dist-test-slave-mz20"
server_key: "c772e368c20d69cfcf6e8498b7d00eef"
server_key_iv: "95ccbbf0a3200585f8fa6eb3c92df5de"
server_key_version: "encryptionkey@0"
I20260620 06:58:25.049816 9005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 06:58:25.053190 9020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:25.054476 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:25.054641 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "a3ea126f86d944ab857ed374c9c639f5"
format_stamp: "Formatted at 2026-06-20 06:58:25 on dist-test-slave-mz20"
server_key: "c772e368c20d69cfcf6e8498b7d00eef"
server_key_iv: "95ccbbf0a3200585f8fa6eb3c92df5de"
server_key_version: "encryptionkey@0"
I20260620 06:58:25.054822 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:25.068533 9005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:25.072315 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:25.072540 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:25.073417 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:25.074733 9005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:25.074802 9005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:25.074890 9005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:25.074942 9005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:25.103996 9005 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45297
I20260620 06:58:25.104043 9132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45297 every 8 connection(s)
I20260620 06:58:25.105242 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:25.106389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9005
I20260620 06:58:25.106594 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:25.107131 388 external_mini_cluster.cc:1428] Setting key ed58c942e82743e5e544aeb29dfa24c5
Jun 20 06:58:25 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:25.125507 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:25.108060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58889 (local address 127.0.97.62:33079)
0620 06:58:25.108236 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:25.108240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:25.109517 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 06:58:25.111440 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:25.111444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:25.111529 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:25.111618 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:25.113096 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.113853 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.115472 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.115928 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.116029 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:25.119347 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:25.119352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:25.119355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:25.119389 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:25.122290 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.123297 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.123303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.123306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.123442 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.123821 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.123824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.123826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.124127 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:25.124234 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:25.125092 (+ 858us) server_negotiation.cc:299] Negotiation successful
0620 06:58:25.125258 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:58:25.126478 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:25.108563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:58889)
0620 06:58:25.109341 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:25.109400 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:25.111178 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:25.111886 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:25.111898 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:25.112388 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:25.112952 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:25.112969 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.113972 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.113975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:25.115327 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:25.115338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.116115 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.116119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:25.116317 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:25.116932 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:25.116953 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:25.119114 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:25.122607 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:25.122617 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:25.122636 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:25.123144 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:25.123544 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:25.123547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:25.123549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:25.123737 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:25.124274 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:25.124280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:25.124649 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 06:58:25.124946 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 06:58:25.125348 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":293,"threads_started":1}
I20260620 06:58:25.127977 9133 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33079
I20260620 06:58:25.128253 9133 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:25.128996 9133 heartbeater.cc:511] Master 127.0.97.62:33079 requested a full tablet report, sending...
I20260620 06:58:25.130340 8677 ts_manager.cc:194] Registered new tserver with Master: a3ea126f86d944ab857ed374c9c639f5 (127.0.97.3:45297)
I20260620 06:58:25.131153 8677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58889
I20260620 06:58:25.134817 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 06:58:25.185398 8866 heartbeater.cc:503] Master 127.0.97.62:33079 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 06:58:25.256888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8738
Jun 20 06:58:25 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:58:25.271921 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:25.257890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52188 (local address 127.0.97.1:39601)
0620 06:58:25.258655 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:25.258662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:25.258685 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:58:25.258742 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:25.258748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:25.258959 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:25.259127 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:25.260369 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.261348 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.262563 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.263103 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.263220 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:25.266875 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:25.266886 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:25.266900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:25.266930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:25.269377 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.269989 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.269996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.269999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.270121 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.270440 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.270444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.270446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.270794 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:25.271071 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:25.271348 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 06:58:25.271597 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":423,"threads_started":1}
I20260620 06:58:25.273087 8861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:25.353750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8738
I20260620 06:58:25.366832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8872
Jun 20 06:58:25 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:58:25.381626 9145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:25.367650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.97.2:43115)
0620 06:58:25.368200 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:25.368207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:25.368232 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:58:25.368297 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:25.368303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:25.368510 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:25.368644 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:25.369800 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.370816 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.372028 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.372562 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.372656 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:25.376634 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:25.376645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:25.376659 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:25.376690 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:25.379304 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.379960 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.379967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.379969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.380095 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.380355 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.380358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.380360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.380692 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:25.380907 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:25.381122 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:58:25.381333 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260620 06:58:25.382899 8995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:25.463404 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8872
I20260620 06:58:25.478344 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9005
Jun 20 06:58:25 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:58:25.493192 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:25.479300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.0.97.3:45297)
0620 06:58:25.479898 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:25.479905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:25.479960 (+ 55us) server_negotiation.cc:407] Connection header received
0620 06:58:25.480120 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:25.480127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:25.480333 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:25.480465 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:25.481604 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.482684 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.483857 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:25.484391 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:25.484478 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:25.488566 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:25.488576 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:25.488591 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:25.488621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:25.491078 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.491636 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.491644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.491647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.491763 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:25.492021 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:25.492024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:25.492026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:25.492345 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:25.492562 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:25.492741 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 06:58:25.492917 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":302,"threads_started":1}
I20260620 06:58:25.494402 9128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:25.574771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9005
I20260620 06:58:25.591399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 8645
I20260620 06:58:25.592494 8707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:25.691212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 8645
I20260620 06:58:25.707175 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--webserver_interface=127.0.97.62
--webserver_port=42165
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:26.014092 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:26.014394 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:26.014511 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:26.021507 9150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:26.021628 9150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:26.021699 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:26.021775 9150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:26.021831 9150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:26.031378 9150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:26.032680 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:26.034034 9150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:26.045173 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.045257 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.045176 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:26.045665 9150 server_base.cc:1034] running on GCE node
I20260620 06:58:26.046525 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:26.047905 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:26.049124 9150 hybrid_clock.cc:648] HybridClock initialized: now 1781938706049086 us; error 29 us; skew 500 ppm
I20260620 06:58:26.049465 9150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:26.052371 9150 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:58:26.053794 9150 webserver.cc:460] Webserver started at http://127.0.97.62:42165/ using document root <none> and password file <none>
I20260620 06:58:26.054433 9150 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:26.054574 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:26.059780 9150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:26.062384 9165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.063722 9150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:26.063863 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "31012f5c86d94e158a1572ec81e7cc36"
format_stamp: "Formatted at 2026-06-20 06:58:23 on dist-test-slave-mz20"
server_key: "ac523c747e7e28d5d9c3313a9a09face"
server_key_iv: "b8526eb247140878bf6db59ce15e2097"
server_key_version: "encryptionkey@0"
I20260620 06:58:26.064754 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:26.080307 9150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:26.084115 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:26.084343 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:26.101305 9150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33079
I20260620 06:58:26.101328 9216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33079 every 8 connection(s)
I20260620 06:58:26.102819 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:26.106591 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9150
I20260620 06:58:26.107336 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39601
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=45015
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:58:26.109530 9217 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:58:26.113396 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Bootstrap starting.
I20260620 06:58:26.131363 9217 log.cc:824] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:26.139654 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:58:26.140115 9217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Bootstrap complete.
I20260620 06:58:26.148609 9217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:26.149411 9217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31012f5c86d94e158a1572ec81e7cc36, State: Initialized, Role: FOLLOWER
I20260620 06:58:26.149992 9217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:26.150205 9217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:26.150302 9217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:26.150446 9217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:58:26.154042 9217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:26.154524 9217 leader_election.cc:302] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31012f5c86d94e158a1572ec81e7cc36; no voters:
I20260620 06:58:26.155216 9217 leader_election.cc:288] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:58:26.155366 9221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:58:26.156360 9221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [term 2 LEADER]: Becoming Leader. State: Replica: 31012f5c86d94e158a1572ec81e7cc36, State: Running, Role: LEADER
I20260620 06:58:26.156991 9221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } }
I20260620 06:58:26.157402 9217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:26.159449 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31012f5c86d94e158a1572ec81e7cc36. Latest consensus state: current_term: 2 leader_uuid: "31012f5c86d94e158a1572ec81e7cc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260620 06:58:26.159475 9222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31012f5c86d94e158a1572ec81e7cc36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31012f5c86d94e158a1572ec81e7cc36" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33079 } } }
I20260620 06:58:26.159794 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:26.159837 9222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:26.167117 9227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:26.173846 9227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:26.175413 9227 catalog_manager.cc:1259] Loaded cluster ID: b2c59527dedb415d9c9ab69e0f6bf28a
I20260620 06:58:26.175604 9227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:26.179342 9227 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:26.180650 9238 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:58:26.180836 9227 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31012f5c86d94e158a1572ec81e7cc36: Loaded TSK: 0
I20260620 06:58:26.182412 9227 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:58:26.432559 9219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:26.432829 9219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:26.432945 9219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:26.440215 9219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:26.440352 9219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:26.440521 9219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:26.450240 9219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45015
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:26.451538 9219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:26.452795 9219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:26.467643 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.467821 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.468005 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:26.468704 9219 server_base.cc:1034] running on GCE node
I20260620 06:58:26.469386 9219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:26.470371 9219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:26.471572 9219 hybrid_clock.cc:648] HybridClock initialized: now 1781938706471523 us; error 64 us; skew 500 ppm
I20260620 06:58:26.471882 9219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:26.474915 9219 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 06:58:26.476415 9219 webserver.cc:460] Webserver started at http://127.0.97.1:45015/ using document root <none> and password file <none>
I20260620 06:58:26.477080 9219 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:26.477212 9219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:26.482440 9219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:58:26.485123 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.486498 9219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:26.486687 9219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "22fefa68552c40e3a28e7a191ae8f461"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "0cfcd8ba80d9fef5d8cdf2974379b720"
server_key_iv: "b59c35bb8a79bc57b5b6aef16432d99e"
server_key_version: "encryptionkey@0"
I20260620 06:58:26.487597 9219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:26.509132 9219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:26.513249 9219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:26.513481 9219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:26.514423 9219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:26.515683 9219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:26.515774 9219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.515875 9219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:26.515942 9219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.545454 9219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39601
I20260620 06:58:26.545531 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39601 every 8 connection(s)
I20260620 06:58:26.546830 9219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:26.547896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9219
I20260620 06:58:26.548573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43115
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=44515
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:26.565946 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:26.550037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:54997)
0620 06:58:26.550865 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:26.550918 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:26.552630 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:26.554624 (+ 1994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:26.554636 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:26.555237 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:26.555770 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.555788 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.557822 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.557826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.559147 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.559161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.560165 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.560172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.560418 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.561066 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:26.561092 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:26.564595 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":320,"threads_started":1}
W20260620 06:58:26.565984 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:26.550031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54997 (local address 127.0.97.62:33079)
0620 06:58:26.553331 (+ 3300us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:26.553343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:26.553380 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:58:26.553693 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:26.553708 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:26.554060 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:26.554419 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:26.556014 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.557659 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.559286 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.559934 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.560100 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.564991 (+ 4891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54997: BlockingRecv error: recv got EOF from 127.0.97.1:54997 (error 108)
Metrics: {"server-negotiator.queue_time_us":3084,"thread_start_us":2881,"threads_started":1}
W20260620 06:58:26.566826 9366 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33079: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:26.575683 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:26.567468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:49875)
0620 06:58:26.567628 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:26.567646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:26.567762 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:26.568172 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:26.568176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:26.568526 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:26.568682 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.568692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.570142 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.570146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.571149 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.571161 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.572078 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.572083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.572206 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.572772 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:26.572790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:26.575445 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:58:26.575898 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:26.567683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49875 (local address 127.0.97.62:33079)
0620 06:58:26.567833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:26.567839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:26.567856 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:58:26.567903 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:26.567907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:26.567971 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:26.568052 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:26.568849 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.569973 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.571267 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.571890 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.572008 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.575678 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49875: BlockingRecv error: recv got EOF from 127.0.97.1:49875 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:26.583775 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:26.576449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:60195)
0620 06:58:26.576589 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:26.576607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:26.576687 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:26.576977 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:26.576979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:26.577283 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:26.577419 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.577426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.578745 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.578749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.579658 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.579669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.580470 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.580474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.580603 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.581120 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:26.581136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:26.583582 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 06:58:26.584038 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:26.576523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60195 (local address 127.0.97.62:33079)
0620 06:58:26.576663 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:26.576668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:26.576686 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:58:26.576778 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:26.576782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:26.576867 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:26.576965 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:26.577528 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.578620 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.579788 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.580357 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.580468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.583803 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60195: BlockingRecv error: recv got EOF from 127.0.97.1:60195 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:58:26.584165 9366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:58:26.864578 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:26.864832 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:26.864948 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:26.872202 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:26.872308 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:26.872442 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:26.881953 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44515
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:26.883208 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:26.884472 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:26.897065 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.897100 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:26.897142 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:26.900022 9370 server_base.cc:1034] running on GCE node
I20260620 06:58:26.900656 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:26.901647 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:26.902853 9370 hybrid_clock.cc:648] HybridClock initialized: now 1781938706902790 us; error 60 us; skew 500 ppm
I20260620 06:58:26.903167 9370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938706, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:26.906131 9370 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 06:58:26.907688 9370 webserver.cc:460] Webserver started at http://127.0.97.2:44515/ using document root <none> and password file <none>
I20260620 06:58:26.908375 9370 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:26.908499 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:26.914099 9370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 06:58:26.917037 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.918424 9370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:26.918610 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "2b2a4f3ceb75410ebb8f4e36a74a896d"
format_stamp: "Formatted at 2026-06-20 06:58:24 on dist-test-slave-mz20"
server_key: "c8776655e9bc2d69a1c5153acc383c35"
server_key_iv: "8512813a16dff0c9fbcaf08bfa809d32"
server_key_version: "encryptionkey@0"
I20260620 06:58:26.919565 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:26.941823 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:26.945766 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:26.945972 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:26.946884 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:26.948192 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:26.948283 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.948359 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:26.948410 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:26.977284 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43115
I20260620 06:58:26.977360 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43115 every 8 connection(s)
I20260620 06:58:26.978766 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:26.980378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9370
I20260620 06:58:26.980871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45297
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33079
--webserver_port=36593
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:26 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:58:27.001538 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:26.986057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46213 (local address 127.0.97.62:33079)
0620 06:58:26.986257 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:26.986264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:26.987523 (+ 1259us) server_negotiation.cc:407] Connection header received
0620 06:58:26.989913 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:26.989917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:26.989985 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:26.990065 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:26.992034 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.992769 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.995012 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.995456 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.995588 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.001356 (+ 5768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46213: BlockingRecv error: recv got EOF from 127.0.97.2:46213 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:58:27.002607 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:26.986529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:46213)
0620 06:58:26.987344 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:26.987404 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:26.989620 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:26.990333 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:26.990344 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:26.991058 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:26.991838 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.991864 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.992921 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.992926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.994868 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:26.994884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:26.995799 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:26.995806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:26.996137 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:26.996875 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:26.996906 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.001195 (+ 4289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
W20260620 06:58:27.003382 9498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33079: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.012357 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.003918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:58461)
0620 06:58:27.004072 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.004090 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.004188 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.004447 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.004451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.004831 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.005008 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.005018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.005980 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.005985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.007454 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.007468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.008111 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.008117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.008288 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.008809 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.008830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.012088 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:58:27.012454 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.004038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58461 (local address 127.0.97.62:33079)
0620 06:58:27.004163 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.004167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.004182 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:58:27.004287 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.004290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.004341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.004412 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.005135 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.005875 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.007578 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.007990 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.008078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.012273 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58461: BlockingRecv error: recv got EOF from 127.0.97.2:58461 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.020984 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.013087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:37541)
0620 06:58:27.013232 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.013249 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.013338 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.013579 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.013583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.013929 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.014099 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.014109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.015042 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.015047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.016332 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.016346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.016949 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.016955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.017106 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.017648 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.017669 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.020703 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 06:58:27.021023 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.013195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37541 (local address 127.0.97.62:33079)
0620 06:58:27.013331 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.013334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.013348 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:58:27.013427 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.013430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.013478 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.013546 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.014202 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.014923 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.016470 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.016835 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.016919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.020872 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37541: BlockingRecv error: recv got EOF from 127.0.97.2:37541 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 06:58:27.021241 9498 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:58:27.299998 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:27.300294 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:27.300431 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:27.307511 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:27.307631 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:27.307780 9500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:27.317301 9500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36593
--tserver_master_addrs=127.0.97.62:33079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:27.318639 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:27.319926 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:27.332405 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:27.332466 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:27.332437 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:27.333216 9500 server_base.cc:1034] running on GCE node
I20260620 06:58:27.333809 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:27.334772 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:27.335963 9500 hybrid_clock.cc:648] HybridClock initialized: now 1781938707335902 us; error 56 us; skew 500 ppm
I20260620 06:58:27.336262 9500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938707, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:27.339146 9500 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 06:58:27.340492 9500 webserver.cc:460] Webserver started at http://127.0.97.3:36593/ using document root <none> and password file <none>
I20260620 06:58:27.341147 9500 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:27.341267 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:27.346458 9500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:27.348979 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:27.350184 9500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:58:27.350335 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "a3ea126f86d944ab857ed374c9c639f5"
format_stamp: "Formatted at 2026-06-20 06:58:25 on dist-test-slave-mz20"
server_key: "c772e368c20d69cfcf6e8498b7d00eef"
server_key_iv: "95ccbbf0a3200585f8fa6eb3c92df5de"
server_key_version: "encryptionkey@0"
I20260620 06:58:27.351248 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:27.367637 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:27.371418 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:27.371654 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:27.372539 9500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:27.373709 9500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:27.373799 9500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:27.373898 9500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:27.373963 9500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:27.403033 9500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45297
I20260620 06:58:27.403095 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45297 every 8 connection(s)
I20260620 06:58:27.404450 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:27.412232 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9500
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:58:27.421131 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.406772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45391 (local address 127.0.97.62:33079)
0620 06:58:27.406960 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.406966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.408211 (+ 1245us) server_negotiation.cc:407] Connection header received
0620 06:58:27.410029 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.410033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.410098 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.410181 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.411892 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.412714 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.414621 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.415135 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.415270 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.420877 (+ 5607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45391: BlockingRecv error: recv got EOF from 127.0.97.3:45391 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:58:27.421659 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.407344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:45391)
0620 06:58:27.408043 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.408082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.409779 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.410454 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.410464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.411041 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.411702 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.411720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.412865 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.412870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.414468 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.414479 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.415342 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.415347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.415542 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.416341 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.416365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.420650 (+ 4285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":296,"threads_started":1}
W20260620 06:58:27.422304 9629 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33079: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.429116 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.413181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.0.97.62:33079)
0620 06:58:27.414885 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.414889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.414905 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:58:27.414941 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.414944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.414995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.415082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.416704 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.417747 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.419216 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.419848 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.419982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.422602 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:27.422614 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:27.422636 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:27.422680 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:27.425880 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:27.426517 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:27.426523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:27.426530 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:27.426754 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:27.427190 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:27.427195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:27.427198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:27.427792 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:27.428077 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:27.428457 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 06:58:27.428828 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1614,"thread_start_us":1519,"threads_started":1}
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.430753 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.422833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:51387)
0620 06:58:27.423047 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.423064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.423140 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.423432 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.423437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.423801 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.423932 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.423940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.425286 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.425290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.426207 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.426219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.427019 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.427023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.427195 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.427684 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.427705 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.430473 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 06:58:27.430894 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.422971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51387 (local address 127.0.97.62:33079)
0620 06:58:27.423137 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.423142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.423159 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:58:27.423226 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.423231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.423298 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.423379 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.424057 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.425143 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.426347 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.426867 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.426987 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.430686 (+ 3699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51387: BlockingRecv error: recv got EOF from 127.0.97.3:51387 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.438406 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.431708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:59785)
0620 06:58:27.431850 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.431864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.431956 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.432269 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.432272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.432521 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.432685 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.432694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.433561 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.433564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.434432 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.434441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.435081 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.435085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.435201 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.435661 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.435676 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.438128 (+ 2452us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:58:27.438627 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.431801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59785 (local address 127.0.97.62:33079)
0620 06:58:27.432015 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.432020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.432034 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:58:27.432074 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.432079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.432146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.432234 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.432798 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.433461 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.434572 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.434980 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.435071 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.438377 (+ 3306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59785: BlockingRecv error: recv got EOF from 127.0.97.3:59785 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260620 06:58:27.438757 9629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:27 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:27.593197 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:27.584793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:50481)
0620 06:58:27.585318 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:27.585333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:27.585463 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:27.585812 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:27.585816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:27.586351 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:27.586606 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.586620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.587809 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.587813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.588850 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:27.588866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.589678 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.589683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:27.589813 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.590358 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:27.590373 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:27.592983 (+ 2610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
W20260620 06:58:27.593344 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:27.584985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50481 (local address 127.0.97.62:33079)
0620 06:58:27.585208 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:27.585215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:27.585411 (+ 196us) server_negotiation.cc:407] Connection header received
0620 06:58:27.585571 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:27.585577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:27.585678 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:27.585761 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:27.586775 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.587682 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.588995 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:27.589504 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:27.589600 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:27.593172 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50481: BlockingRecv error: recv got EOF from 127.0.97.1:50481 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:28.030179 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:28.021852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:54747)
0620 06:58:28.022347 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:28.022368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:28.022515 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:28.022858 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:28.022863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:28.023361 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:28.023545 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.023554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.024511 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.024517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.025466 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.025478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.026195 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.026201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.026360 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.027007 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:28.027029 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:28.029962 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
W20260620 06:58:28.030356 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:28.021967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54747 (local address 127.0.97.62:33079)
0620 06:58:28.022180 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:28.022186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:28.022468 (+ 282us) server_negotiation.cc:407] Connection header received
0620 06:58:28.022651 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:28.022657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:28.022722 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:28.022834 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:28.023693 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.024366 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.025604 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.026027 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.026133 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.030141 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54747: BlockingRecv error: recv got EOF from 127.0.97.2:54747 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:28.446906 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:28.439333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:50675)
0620 06:58:28.439788 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:28.439804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:28.439909 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:28.440143 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:28.440146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:28.440556 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:28.440740 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.440748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.441675 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.441680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.442628 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.442640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.443312 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.443317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.443444 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.443967 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:28.443983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:28.446681 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
W20260620 06:58:28.447069 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:28.439386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50675 (local address 127.0.97.62:33079)
0620 06:58:28.439553 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:28.439557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:28.439867 (+ 310us) server_negotiation.cc:407] Connection header received
0620 06:58:28.439996 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:28.440000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:28.440056 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:28.440130 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:28.440847 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.441540 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.442754 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.443127 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.443213 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.446866 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50675: BlockingRecv error: recv got EOF from 127.0.97.3:50675 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:28 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:28.601630 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:28.594055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:36925)
0620 06:58:28.594584 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:28.594600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:28.594711 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:28.595022 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:28.595025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:28.595431 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:28.595573 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.595581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.596582 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.596588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.597531 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:28.597543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.598164 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.598169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:28.598276 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.598854 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:28.598872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:28.601384 (+ 2512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":283,"threads_started":1}
W20260620 06:58:28.601791 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:28.594167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36925 (local address 127.0.97.62:33079)
0620 06:58:28.594334 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:28.594338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:28.594672 (+ 334us) server_negotiation.cc:407] Connection header received
0620 06:58:28.594804 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:28.594808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:28.594864 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:28.594941 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:28.595709 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.596441 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.597657 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:28.598042 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:28.598128 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:28.601580 (+ 3452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36925: BlockingRecv error: recv got EOF from 127.0.97.1:36925 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:29.038710 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:29.031058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:59969)
0620 06:58:29.031619 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:29.031640 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:29.031757 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:29.032027 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:29.032030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:29.032394 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:29.032535 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.032543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.033459 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.033462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.034353 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.034364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.035065 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.035069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.035175 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.035697 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:29.035713 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:29.038344 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
W20260620 06:58:29.038791 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:29.031162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59969 (local address 127.0.97.62:33079)
0620 06:58:29.031305 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:29.031310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:29.031703 (+ 393us) server_negotiation.cc:407] Connection header received
0620 06:58:29.031844 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:29.031847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:29.031906 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:29.031981 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:29.032640 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.033354 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.034480 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.034941 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.035042 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.038586 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59969: BlockingRecv error: recv got EOF from 127.0.97.2:59969 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:29.458585 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:29.447756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:42815)
0620 06:58:29.448243 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:29.448262 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:29.448390 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:29.448709 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:29.448714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:29.449199 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:29.449372 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.449381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.450317 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.450321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.451247 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.451259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.451937 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.451941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.452070 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.455544 (+ 3474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:29.455564 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:29.458298 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":265,"threads_started":1}
W20260620 06:58:29.458878 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:29.447914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42815 (local address 127.0.97.62:33079)
0620 06:58:29.448105 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:29.448111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:29.448347 (+ 236us) server_negotiation.cc:407] Connection header received
0620 06:58:29.448488 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:29.448493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:29.448568 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:29.448658 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:29.449486 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.450194 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.451389 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.451807 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.451926 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.458527 (+ 6601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42815: BlockingRecv error: recv got EOF from 127.0.97.3:42815 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:29 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:29.610219 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:29.602664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:54553)
0620 06:58:29.603117 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:29.603136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:29.603252 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:29.603606 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:29.603609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:29.603918 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:29.604084 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.604092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.605024 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.605027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.605922 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:29.605933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.606611 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.606616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:29.606747 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.607355 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:29.607372 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:29.609862 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":233,"threads_started":1}
W20260620 06:58:29.610285 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:29.602781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54553 (local address 127.0.97.62:33079)
0620 06:58:29.602981 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:29.602985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:29.603210 (+ 225us) server_negotiation.cc:407] Connection header received
0620 06:58:29.603396 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:29.603401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:29.603463 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:29.603547 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:29.604196 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.604913 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.606030 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:29.606415 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:29.606504 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:29.610071 (+ 3567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54553: BlockingRecv error: recv got EOF from 127.0.97.1:54553 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:30.047833 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:30.039669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:40053)
0620 06:58:30.040188 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:30.040207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:30.040338 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:30.040672 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:30.040677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:30.041008 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:30.041164 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.041172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.042199 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.042205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.043131 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.043145 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.043811 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.043817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.043978 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.044667 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:30.044686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:30.047463 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":281,"threads_started":1}
W20260620 06:58:30.047899 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:30.039782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40053 (local address 127.0.97.62:33079)
0620 06:58:30.040013 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:30.040019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:30.040291 (+ 272us) server_negotiation.cc:407] Connection header received
0620 06:58:30.040448 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:30.040452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:30.040511 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:30.040594 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:30.041310 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.042059 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.043270 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.043677 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.043782 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.047672 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40053: BlockingRecv error: recv got EOF from 127.0.97.2:40053 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:30.467131 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:30.459416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:35427)
0620 06:58:30.459916 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:30.459931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:30.460083 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:30.460417 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:30.460422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:30.460760 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:30.460918 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.460926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.461891 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.461897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.462810 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.462821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.463470 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.463475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.463631 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.464155 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:30.464172 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:30.466858 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
W20260620 06:58:30.467289 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:30.459500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35427 (local address 127.0.97.62:33079)
0620 06:58:30.459742 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:30.459748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:30.460013 (+ 265us) server_negotiation.cc:407] Connection header received
0620 06:58:30.460201 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:30.460206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:30.460290 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:30.460368 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:30.461058 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.461771 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.462943 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.463341 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.463444 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.467078 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35427: BlockingRecv error: recv got EOF from 127.0.97.3:35427 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:30 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:30.619191 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:30.611177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:37059)
0620 06:58:30.611655 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:30.611670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:30.611822 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:30.612104 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:30.612109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:30.612467 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:30.612638 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.612647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.613657 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.613663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.614718 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:30.614730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.615384 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.615390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:30.615558 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.616112 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:30.616128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:30.618959 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":263,"threads_started":1}
W20260620 06:58:30.619385 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:30.611296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37059 (local address 127.0.97.62:33079)
0620 06:58:30.611525 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:30.611531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:30.611755 (+ 224us) server_negotiation.cc:407] Connection header received
0620 06:58:30.611929 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:30.611932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:30.611988 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:30.612066 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:30.612769 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.613537 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.614851 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:30.615255 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:30.615359 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:30.619168 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37059: BlockingRecv error: recv got EOF from 127.0.97.1:37059 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:31.056934 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:31.048763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:58215)
0620 06:58:31.049245 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:31.049267 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:31.049399 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:31.049700 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:31.049704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:31.050063 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:31.050216 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.050225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.051233 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.051238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.052179 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.052189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.052852 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.052858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.053035 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.053579 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:31.053595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:31.056597 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":264,"threads_started":1}
W20260620 06:58:31.056993 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:31.048922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58215 (local address 127.0.97.62:33079)
0620 06:58:31.049140 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:31.049146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:31.049353 (+ 207us) server_negotiation.cc:407] Connection header received
0620 06:58:31.049509 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:31.049514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:31.049577 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:31.049656 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:31.050355 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.051109 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.052313 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.052721 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.052829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.056791 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58215: BlockingRecv error: recv got EOF from 127.0.97.2:58215 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:31.481506 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:31.468102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:55287)
0620 06:58:31.468883 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:31.468926 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:31.469121 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:31.469793 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:31.469799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:31.470253 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:31.470440 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.470453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.471914 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.471920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.473434 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.473449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.474444 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.474450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.474646 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.475368 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:31.475392 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:31.481183 (+ 5791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":756,"threads_started":1}
W20260620 06:58:31.481685 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:31.468186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55287 (local address 127.0.97.62:33079)
0620 06:58:31.468858 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:31.468864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:31.469348 (+ 484us) server_negotiation.cc:407] Connection header received
0620 06:58:31.469428 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:31.469433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:31.469509 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:31.469598 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:31.470611 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.471729 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.473596 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.474184 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.474278 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.481426 (+ 7148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55287: BlockingRecv error: recv got EOF from 127.0.97.3:55287 (error 108)
Metrics: {"server-negotiator.queue_time_us":555}
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:31 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:31.627971 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:31.620118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:37207)
0620 06:58:31.620656 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:31.620679 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:31.620785 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:31.621094 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:31.621099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:31.621454 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:31.621612 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.621621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.622695 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.622699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.623673 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:31.623685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.624336 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.624341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:31.624487 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.625021 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:31.625037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:31.627726 (+ 2689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
W20260620 06:58:31.628077 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:31.620231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37207 (local address 127.0.97.62:33079)
0620 06:58:31.620421 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:31.620428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:31.620764 (+ 336us) server_negotiation.cc:407] Connection header received
0620 06:58:31.620902 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:31.620908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:31.620968 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:31.621044 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:31.621788 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.622567 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.623798 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:31.624210 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:31.624310 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:31.627897 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37207: BlockingRecv error: recv got EOF from 127.0.97.1:37207 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:32.065620 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:32.057904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:36999)
0620 06:58:32.058398 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:32.058418 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:32.058529 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:32.058914 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:32.058919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:32.059395 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:32.059571 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.059583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.060745 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.060751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.061655 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.061666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.062237 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.062241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.062356 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.062904 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:32.062921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:32.065396 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":215,"threads_started":1}
W20260620 06:58:32.065798 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:32.058027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36999 (local address 127.0.97.62:33079)
0620 06:58:32.058182 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:32.058186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:32.058488 (+ 302us) server_negotiation.cc:407] Connection header received
0620 06:58:32.058681 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:32.058686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:32.058760 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:32.058860 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:32.059722 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.060598 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.061757 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.062137 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.062220 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.065572 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36999: BlockingRecv error: recv got EOF from 127.0.97.2:36999 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:32.490370 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:32.482400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:36059)
0620 06:58:32.482940 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:32.482956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:32.483062 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:32.483313 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:32.483316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:32.483734 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:32.483926 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.483937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.484929 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.484933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.485821 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.485832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.486503 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.486508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.486671 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.487282 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:32.487299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:32.490123 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
W20260620 06:58:32.490562 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:32.482449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36059 (local address 127.0.97.62:33079)
0620 06:58:32.482643 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:32.482647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:32.483020 (+ 373us) server_negotiation.cc:407] Connection header received
0620 06:58:32.483142 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:32.483145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:32.483199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:32.483276 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:32.484059 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.484767 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.485937 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.486361 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.486452 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.490335 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36059: BlockingRecv error: recv got EOF from 127.0.97.3:36059 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:32 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:32.636603 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:32.628889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:43685)
0620 06:58:32.629346 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:32.629363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:32.629477 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:32.629757 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:32.629761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:32.630150 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:32.630313 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.630321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.631456 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.631461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.632619 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:32.632630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.633278 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.633282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:32.633397 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.633951 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:32.633967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:32.636374 (+ 2407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":262,"threads_started":1}
W20260620 06:58:32.636763 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:32.628979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43685 (local address 127.0.97.62:33079)
0620 06:58:32.629191 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:32.629198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:32.629432 (+ 234us) server_negotiation.cc:407] Connection header received
0620 06:58:32.629563 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:32.629567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:32.629626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:32.629704 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:32.630418 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.631324 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.632736 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:32.633159 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:32.633246 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:32.636541 (+ 3295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43685: BlockingRecv error: recv got EOF from 127.0.97.1:43685 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:33.074755 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:33.066604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:50093)
0620 06:58:33.067146 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:33.067168 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:33.067301 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:33.067580 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:33.067583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:33.067972 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:33.068181 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.068190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.069205 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.069208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.070115 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.070126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.070798 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.070804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.070929 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.071603 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:33.071625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:33.074443 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":337,"threads_started":1}
W20260620 06:58:33.074928 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:33.066667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50093 (local address 127.0.97.62:33079)
0620 06:58:33.066843 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:33.066850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:33.067239 (+ 389us) server_negotiation.cc:407] Connection header received
0620 06:58:33.067399 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:33.067403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:33.067463 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:33.067548 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:33.068296 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.069083 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.070237 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.070655 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.070743 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.074700 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50093: BlockingRecv error: recv got EOF from 127.0.97.2:50093 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:33.499325 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:33.491358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:41761)
0620 06:58:33.492124 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:33.492140 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:33.492256 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:33.492530 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:33.492533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:33.492904 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:33.493083 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.493092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.494093 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.494097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.495041 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.495054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.495710 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.495714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.495831 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.496427 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:33.496444 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:33.499089 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":545,"threads_started":1}
W20260620 06:58:33.499534 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:33.491544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41761 (local address 127.0.97.62:33079)
0620 06:58:33.491752 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:33.491759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:33.492222 (+ 463us) server_negotiation.cc:407] Connection header received
0620 06:58:33.492348 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:33.492352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:33.492415 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:33.492503 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:33.493216 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.493969 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.495158 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.495580 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.495678 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.499322 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41761: BlockingRecv error: recv got EOF from 127.0.97.3:41761 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:33 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:33.645395 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:33.637516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:58243)
0620 06:58:33.638058 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:33.638073 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:33.638182 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:33.638492 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:33.638495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:33.638886 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:33.639057 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.639066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.640023 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.640026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.640919 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:33.640929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.641826 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.641832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:33.641943 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.642586 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:33.642602 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:33.645166 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":237,"threads_started":1}
W20260620 06:58:33.645529 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:33.637598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58243 (local address 127.0.97.62:33079)
0620 06:58:33.637816 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:33.637821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:33.638151 (+ 330us) server_negotiation.cc:407] Connection header received
0620 06:58:33.638271 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:33.638275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:33.638333 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:33.638421 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:33.639218 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.639918 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.641059 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:33.641664 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:33.641768 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:33.645340 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58243: BlockingRecv error: recv got EOF from 127.0.97.1:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:34.083602 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:34.075580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:50607)
0620 06:58:34.076073 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:34.076102 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:34.076216 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:34.076473 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:34.076476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:34.076839 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:34.077022 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.077032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.078203 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.078210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.079436 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.079449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.080054 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.080058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.080187 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.080707 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:34.080722 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:34.083358 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":212,"threads_started":1}
W20260620 06:58:34.083817 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:34.075683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50607 (local address 127.0.97.62:33079)
0620 06:58:34.075923 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:34.075929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:34.076172 (+ 243us) server_negotiation.cc:407] Connection header received
0620 06:58:34.076319 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:34.076323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:34.076381 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:34.076458 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:34.077154 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.078072 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.079561 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.079945 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.080043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.083560 (+ 3517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50607: BlockingRecv error: recv got EOF from 127.0.97.2:50607 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:34.511029 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:34.500178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:37075)
0620 06:58:34.500750 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:34.500766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:34.500894 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:34.501195 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:34.501200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:34.501582 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:34.501751 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.501759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.502762 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.502766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.503669 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.503680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.504366 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.504372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.504520 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.508206 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:34.508223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:34.510791 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":253,"threads_started":1}
W20260620 06:58:34.511221 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:34.500226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37075 (local address 127.0.97.62:33079)
0620 06:58:34.500378 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:34.500382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:34.500851 (+ 469us) server_negotiation.cc:407] Connection header received
0620 06:58:34.501001 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:34.501007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:34.501068 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:34.501145 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:34.501879 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.502640 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.503796 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.504225 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.504331 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.511035 (+ 6704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37075: BlockingRecv error: recv got EOF from 127.0.97.3:37075 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:34 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:34.655638 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:34.646381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:39155)
0620 06:58:34.646969 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:34.646987 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:34.647128 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:34.647513 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:34.647522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:34.647985 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:34.648164 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.648173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.649299 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.649303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.650260 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:34.650274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.651126 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.651132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:34.651269 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.651964 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:34.651986 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:34.655191 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
W20260620 06:58:34.655849 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:34.646653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39155 (local address 127.0.97.62:33079)
0620 06:58:34.646843 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:34.646848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:34.647096 (+ 248us) server_negotiation.cc:407] Connection header received
0620 06:58:34.647228 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:34.647232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:34.647302 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:34.647432 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:34.648334 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.649163 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.650427 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:34.650950 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:34.651045 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:34.655551 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39155: BlockingRecv error: recv got EOF from 127.0.97.1:39155 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:35.094719 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:35.084689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:49087)
0620 06:58:35.085282 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:35.085304 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:35.085442 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:35.085855 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:35.085859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:35.086368 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:35.086606 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.086624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.088082 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.088097 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.089471 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.089486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.090313 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.090319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.090594 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.091319 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:35.091340 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:35.094373 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":332,"threads_started":1}
W20260620 06:58:35.094877 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:35.084801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49087 (local address 127.0.97.62:33079)
0620 06:58:35.084974 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:35.084979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:35.085385 (+ 406us) server_negotiation.cc:407] Connection header received
0620 06:58:35.085590 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:35.085595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:35.085669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:35.085761 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:35.086796 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.087892 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.089637 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.090137 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.090244 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.094669 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49087: BlockingRecv error: recv got EOF from 127.0.97.2:49087 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:35.521762 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:35.512181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:58591)
0620 06:58:35.512821 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:35.512841 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:35.512995 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:35.513349 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:35.513354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:35.513876 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:35.514076 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.514087 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.515455 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.515462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.516651 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.516672 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.517493 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.517499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.517636 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.518346 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:35.518366 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:35.521478 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":359,"threads_started":1}
W20260620 06:58:35.521973 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:35.512251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58591 (local address 127.0.97.62:33079)
0620 06:58:35.512451 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:35.512456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:35.512941 (+ 485us) server_negotiation.cc:407] Connection header received
0620 06:58:35.513105 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:35.513109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:35.513183 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:35.513279 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:35.514266 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.515237 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.516802 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.517278 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.517376 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.521711 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58591: BlockingRecv error: recv got EOF from 127.0.97.3:58591 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:35 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:35.665453 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:35.656798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:60099)
0620 06:58:35.657394 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:35.657410 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:35.657575 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:35.657967 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:35.657971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:35.658431 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:35.658641 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.658651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.659730 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.659736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.660671 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:35.660682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.661402 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.661408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:35.661567 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.662155 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:35.662174 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:35.665158 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":329,"threads_started":1}
W20260620 06:58:35.665584 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:35.656875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60099 (local address 127.0.97.62:33079)
0620 06:58:35.657126 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:35.657133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:35.657494 (+ 361us) server_negotiation.cc:407] Connection header received
0620 06:58:35.657748 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:35.657752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:35.657825 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:35.657912 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:35.658817 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.659590 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.660804 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:35.661247 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:35.661360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:35.665381 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60099: BlockingRecv error: recv got EOF from 127.0.97.1:60099 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:36.103941 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:36.095777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:59381)
0620 06:58:36.096275 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:36.096295 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:36.096449 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:36.096747 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:36.096751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:36.097150 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:36.097332 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.097341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.098296 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.098300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.099273 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.099284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.099968 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.099975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.100130 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.100650 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:36.100668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:36.103605 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
W20260620 06:58:36.104039 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:36.095821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59381 (local address 127.0.97.62:33079)
0620 06:58:36.096037 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:36.096043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:36.096373 (+ 330us) server_negotiation.cc:407] Connection header received
0620 06:58:36.096558 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:36.096563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:36.096628 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:36.096702 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:36.097465 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.098171 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.099400 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.099820 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.099932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.103796 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59381: BlockingRecv error: recv got EOF from 127.0.97.2:59381 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:36.531610 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:36.522854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:46441)
0620 06:58:36.523359 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:36.523377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:36.523510 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:36.523901 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:36.523905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:36.524309 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:36.524537 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.524546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.525534 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.525538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.526519 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.526531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.527301 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.527305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.527472 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.528108 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:36.528126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:36.531323 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":238,"threads_started":1}
W20260620 06:58:36.531728 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:36.522929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46441 (local address 127.0.97.62:33079)
0620 06:58:36.523136 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:36.523142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:36.523464 (+ 322us) server_negotiation.cc:407] Connection header received
0620 06:58:36.523659 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:36.523664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:36.523736 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:36.523816 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:36.524689 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.525410 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.526699 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.527151 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.527261 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.531566 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46441: BlockingRecv error: recv got EOF from 127.0.97.3:46441 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:36 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:36.674729 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:36.666532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:39671)
0620 06:58:36.667005 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:36.667019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:36.667127 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:36.667459 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:36.667462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:36.667852 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:36.668015 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.668024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.669186 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.669190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.670175 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:36.670188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.670881 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.670885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:36.671004 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.671523 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:36.671541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:36.674468 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
W20260620 06:58:36.674916 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:36.666639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39671 (local address 127.0.97.62:33079)
0620 06:58:36.666855 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:36.666861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:36.667106 (+ 245us) server_negotiation.cc:407] Connection header received
0620 06:58:36.667237 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:36.667243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:36.667338 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:36.667418 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:36.668160 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.669039 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.670292 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:36.670752 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:36.670842 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:36.674710 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39671: BlockingRecv error: recv got EOF from 127.0.97.1:39671 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:37.112787 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:37.104928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:60323)
0620 06:58:37.105417 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:37.105437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:37.105557 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:37.105870 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:37.105874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:37.106264 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:37.106401 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.106409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.107413 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.107419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.108328 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.108340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.109040 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.109046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.109197 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.109766 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:37.109784 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:37.112578 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
W20260620 06:58:37.112969 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:37.105047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60323 (local address 127.0.97.62:33079)
0620 06:58:37.105225 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:37.105231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:37.105522 (+ 291us) server_negotiation.cc:407] Connection header received
0620 06:58:37.105665 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:37.105670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:37.105736 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:37.105815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:37.106572 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.107293 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.108455 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.108889 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.108998 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.112755 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60323: BlockingRecv error: recv got EOF from 127.0.97.2:60323 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:37.540694 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:37.532685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:45313)
0620 06:58:37.533188 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:37.533204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:37.533337 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:37.533654 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:37.533659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:37.534046 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:37.534185 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.534194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.535271 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.535276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.536301 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.536316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.537032 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.537037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.537169 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.537793 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:37.537810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:37.540422 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":282,"threads_started":1}
W20260620 06:58:37.540868 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:37.532884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45313 (local address 127.0.97.62:33079)
0620 06:58:37.533124 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:37.533131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:37.533286 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:58:37.533453 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:37.533457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:37.533519 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:37.533599 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:37.534336 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.535132 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.536458 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.536877 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.537001 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.540634 (+ 3633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45313: BlockingRecv error: recv got EOF from 127.0.97.3:45313 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:37 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:37.684741 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:37.675725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:54847)
0620 06:58:37.676270 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:37.676286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:37.676437 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:37.676751 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:37.676756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:37.677269 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:37.677489 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.677499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.678589 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.678593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.679549 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:37.679562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.680267 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.680273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:37.680446 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.681128 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:37.681147 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:37.684394 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":294,"threads_started":1}
W20260620 06:58:37.684855 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:37.675797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54847 (local address 127.0.97.62:33079)
0620 06:58:37.676031 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:37.676038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:37.676376 (+ 338us) server_negotiation.cc:407] Connection header received
0620 06:58:37.676553 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:37.676559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:37.676627 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:37.676709 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:37.677632 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.678437 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.679682 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:37.680123 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:37.680230 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:37.684631 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54847: BlockingRecv error: recv got EOF from 127.0.97.1:54847 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:38.122386 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:38.113697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:43851)
0620 06:58:38.114306 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:38.114329 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:38.114464 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:38.114789 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:38.114793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:38.115226 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:38.115445 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.115454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.116497 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.116500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.117448 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.117460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.118145 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.118149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.118283 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.119014 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:38.119039 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:38.122102 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":296,"threads_started":1}
W20260620 06:58:38.122514 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:38.113796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43851 (local address 127.0.97.62:33079)
0620 06:58:38.113992 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:38.113997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:38.114403 (+ 406us) server_negotiation.cc:407] Connection header received
0620 06:58:38.114589 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:38.114595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:38.114670 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:38.114760 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:38.115570 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.116383 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.117588 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.118020 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.118113 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.122293 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43851: BlockingRecv error: recv got EOF from 127.0.97.2:43851 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:38.551057 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:38.541699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:43485)
0620 06:58:38.542280 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:38.542298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:38.542452 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:38.542822 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:38.542826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:38.543347 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:38.543552 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.543561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.544600 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.544606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.545561 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.545572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.546257 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.546263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.546436 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.547224 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:38.547244 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:38.550786 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":340,"threads_started":1}
W20260620 06:58:38.551247 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:38.541833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43485 (local address 127.0.97.62:33079)
0620 06:58:38.542038 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:38.542046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:38.542393 (+ 347us) server_negotiation.cc:407] Connection header received
0620 06:58:38.542615 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:38.542619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:38.542688 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:38.542773 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:38.543694 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.544474 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.545694 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.546111 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.546243 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.551036 (+ 4793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43485: BlockingRecv error: recv got EOF from 127.0.97.3:43485 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:38 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:38.693917 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:38.685779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:50893)
0620 06:58:38.686303 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:38.686319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:38.686439 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:38.686839 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:38.686842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:38.687252 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:38.687425 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.687433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.688453 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.688457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.689362 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:38.689372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.690072 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.690075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:38.690214 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.690934 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:38.690953 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:38.693674 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":255,"threads_started":1}
W20260620 06:58:38.694097 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:38.685785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50893 (local address 127.0.97.62:33079)
0620 06:58:38.685969 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:38.685973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:38.686390 (+ 417us) server_negotiation.cc:407] Connection header received
0620 06:58:38.686630 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:38.686634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:38.686707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:38.686795 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:38.687550 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.688332 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.689482 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:38.689957 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:38.690052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:38.693854 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50893: BlockingRecv error: recv got EOF from 127.0.97.1:50893 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:39.131106 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:39.123384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:33301)
0620 06:58:39.123926 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:39.123947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:39.124069 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:39.124345 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:39.124349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:39.124751 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:39.124921 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.124930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.125843 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.125846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.126826 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.126838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.127465 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.127468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.127596 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.128191 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:39.128209 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:39.130867 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":266,"threads_started":1}
W20260620 06:58:39.131278 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:39.123440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33301 (local address 127.0.97.62:33079)
0620 06:58:39.123630 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:39.123637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:39.124052 (+ 415us) server_negotiation.cc:407] Connection header received
0620 06:58:39.124165 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:39.124169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:39.124229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:39.124298 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:39.125059 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.125737 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.126945 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.127348 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.127436 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.131063 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33301: BlockingRecv error: recv got EOF from 127.0.97.2:33301 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:39.562659 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:39.551999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:48049)
0620 06:58:39.552474 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:39.552490 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:39.552624 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:39.552956 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:39.552961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:39.553350 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:39.553511 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.553520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.554495 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.554499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.555698 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.555710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.556373 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.556379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.556538 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.559448 (+ 2910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:39.559467 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:39.562315 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
W20260620 06:58:39.562767 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:39.552040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48049 (local address 127.0.97.62:33079)
0620 06:58:39.552222 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:39.552228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:39.552571 (+ 343us) server_negotiation.cc:407] Connection header received
0620 06:58:39.552734 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:39.552740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:39.552819 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:39.552897 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:39.553644 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.554360 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.555827 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.556242 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.556350 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.562520 (+ 6170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48049: BlockingRecv error: recv got EOF from 127.0.97.3:48049 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:39 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:39.703122 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:39.694898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:39123)
0620 06:58:39.695396 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:39.695411 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:39.695524 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:39.695795 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:39.695799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:39.696152 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:39.696327 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.696335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.697295 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.697299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.698333 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:39.698348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.699539 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.699546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:39.699700 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.700214 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:39.700230 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:39.702898 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":245,"threads_started":1}
W20260620 06:58:39.703330 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:39.694935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39123 (local address 127.0.97.62:33079)
0620 06:58:39.695103 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:39.695108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:39.695478 (+ 370us) server_negotiation.cc:407] Connection header received
0620 06:58:39.695622 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:39.695626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:39.695684 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:39.695766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:39.696451 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.697179 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.698492 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:39.699118 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:39.699260 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:39.703106 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39123: BlockingRecv error: recv got EOF from 127.0.97.1:39123 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:40.139863 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:40.131956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:49503)
0620 06:58:40.132443 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:40.132461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:40.132585 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:40.132872 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:40.132877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:40.133252 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:40.133440 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.133449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.134472 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.134476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.135421 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.135433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.136172 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.136178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.136325 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.136867 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:40.136883 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:40.139553 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":227,"threads_started":1}
W20260620 06:58:40.139979 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:40.132133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49503 (local address 127.0.97.62:33079)
0620 06:58:40.132321 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:40.132328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:40.132536 (+ 208us) server_negotiation.cc:407] Connection header received
0620 06:58:40.132688 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:40.132693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:40.132753 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:40.132829 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:40.133577 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.134327 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.135558 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.136003 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.136109 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.139753 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49503: BlockingRecv error: recv got EOF from 127.0.97.2:49503 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:40.571611 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:40.563576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:38715)
0620 06:58:40.564091 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:40.564107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:40.564217 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:40.564508 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:40.564511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:40.564888 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:40.565097 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.565106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.566071 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.566075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.567085 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.567099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.567822 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.567829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.568029 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.568638 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:40.568656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:40.571374 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":249,"threads_started":1}
W20260620 06:58:40.571765 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:40.563646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38715 (local address 127.0.97.62:33079)
0620 06:58:40.563802 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:40.563806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:40.564172 (+ 366us) server_negotiation.cc:407] Connection header received
0620 06:58:40.564303 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:40.564306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:40.564371 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:40.564447 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:40.565203 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.565947 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.567211 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.567625 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.567714 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.571589 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38715: BlockingRecv error: recv got EOF from 127.0.97.3:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:40 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:40.711885 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:40.703998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:47901)
0620 06:58:40.704602 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:40.704620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:40.704744 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:40.705075 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:40.705079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:40.705440 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:40.705609 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.705617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.706683 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.706689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.707576 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:40.707587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.708220 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.708226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:40.708380 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.708942 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:40.708960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:40.711547 (+ 2587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":389,"threads_started":1}
W20260620 06:58:40.711980 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:40.704054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47901 (local address 127.0.97.62:33079)
0620 06:58:40.704283 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:40.704287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:40.704701 (+ 414us) server_negotiation.cc:407] Connection header received
0620 06:58:40.704851 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:40.704855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:40.704944 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:40.705023 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:40.705746 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.706519 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.707707 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:40.708092 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:40.708195 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:40.711752 (+ 3557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47901: BlockingRecv error: recv got EOF from 127.0.97.1:47901 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:41.148617 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:41.140847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:52959)
0620 06:58:41.141301 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:41.141322 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:41.141445 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:41.141739 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:41.141742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:41.142102 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:41.142249 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.142257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.143364 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.143368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.144450 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.144463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.145134 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.145139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.145252 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.145801 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:41.145818 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:41.148378 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":254,"threads_started":1}
W20260620 06:58:41.148780 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:41.140939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52959 (local address 127.0.97.62:33079)
0620 06:58:41.141161 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:41.141166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:41.141433 (+ 267us) server_negotiation.cc:407] Connection header received
0620 06:58:41.141536 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:41.141540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:41.141603 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:41.141687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:41.142412 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.143231 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.144568 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.144989 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.145108 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.148552 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52959: BlockingRecv error: recv got EOF from 127.0.97.2:52959 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:41.579985 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:41.572535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.3:36207)
0620 06:58:41.572967 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:41.572982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:41.573115 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:41.573361 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:41.573364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:41.573715 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:41.573860 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.573868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.574810 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.574814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.575696 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.575706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.576346 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.576350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.576461 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.577021 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:41.577038 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:41.579660 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":236,"threads_started":1}
W20260620 06:58:41.580088 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:41.572624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36207 (local address 127.0.97.62:33079)
0620 06:58:41.572782 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:41.572786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:41.573044 (+ 258us) server_negotiation.cc:407] Connection header received
0620 06:58:41.573201 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:41.573204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:41.573260 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:41.573335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:41.573963 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.574704 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.575814 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.576238 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.576327 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.579877 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36207: BlockingRecv error: recv got EOF from 127.0.97.3:36207 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:41 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:41.720983 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:41.713052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.1:54403)
0620 06:58:41.713502 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:41.713517 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:41.713644 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:41.713917 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:41.713920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:41.714337 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:41.714587 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.714602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.715891 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.715895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.716795 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:41.716806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.717382 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.717386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:41.717496 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.718026 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:41.718043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:41.720739 (+ 2696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":255,"threads_started":1}
W20260620 06:58:41.721105 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:41.713179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54403 (local address 127.0.97.62:33079)
0620 06:58:41.713385 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:41.713391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:41.713584 (+ 193us) server_negotiation.cc:407] Connection header received
0620 06:58:41.713735 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:41.713739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:41.713813 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:41.713891 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:41.714760 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.715768 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.716896 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:41.717278 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:41.717363 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:41.720922 (+ 3559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54403: BlockingRecv error: recv got EOF from 127.0.97.1:54403 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:58:42 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:58:42 dist-test-slave-mz20 krb5kdc[8632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:58:42.158113 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:42.149649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33079 (local address 127.0.97.2:47413)
0620 06:58:42.150132 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:42.150151 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:42.150268 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:42.150530 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:42.150554 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:42.150957 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:42.151120 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:42.151128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.152250 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.152253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:42.153297 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:42.153309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.154112 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.154120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:42.154266 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:42.154901 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:42.154918 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:42.157678 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33079: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
W20260620 06:58:42.158151 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:42.149773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47413 (local address 127.0.97.62:33079)
0620 06:58:42.149925 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:42.149930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:42.150248 (+ 318us) server_negotiation.cc:407] Connection header received
0620 06:58:42.150358 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:42.150362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:42.150422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:42.150504 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:42.151239 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.152098 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.153431 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.153951 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.154088 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:42.157903 (+ 3815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47413: BlockingRecv error: recv got EOF from 127.0.97.2:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:58:42.416446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9219
I20260620 06:58:42.428579 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:42.417580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.0.97.1:39601)
0620 06:58:42.418200 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:42.418207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:42.418236 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:58:42.418303 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:42.418310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:42.418532 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:42.418752 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:42.420214 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.421249 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.422644 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.423254 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.423357 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:42.425722 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:42.425732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:42.425746 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:42.425781 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:42.428114 (+ 2333us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 06:58:42.428357 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42966: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":297,"threads_started":1}
W20260620 06:58:42.428671 9679 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42966: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 06:58:42.428797 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9219: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39601: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 06:58:42.428987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9219
I20260620 06:58:42.442606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9370
I20260620 06:58:42.454481 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:42.443610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.0.97.2:43115)
0620 06:58:42.444269 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:42.444276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:42.444306 (+ 30us) server_negotiation.cc:407] Connection header received
0620 06:58:42.444364 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:42.444371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:42.444581 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:42.444745 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:42.446044 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.447329 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.448578 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.449142 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.449265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:42.451649 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:42.451658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:42.451673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:42.451707 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:42.453985 (+ 2278us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 06:58:42.454245 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42074: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":368,"threads_started":1}
W20260620 06:58:42.454602 9680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42074: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 06:58:42.454808 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9370: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43115: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 06:58:42.454973 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9370
I20260620 06:58:42.467869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9500
I20260620 06:58:42.479197 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:42.468848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.97.3:45297)
0620 06:58:42.469390 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:42.469397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:42.469421 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:58:42.469476 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:42.469483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:42.469684 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:42.469831 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:42.471047 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.472101 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.473306 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:42.473871 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:42.473973 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:42.476339 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:42.476347 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:42.476360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:42.476401 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:42.478721 (+ 2320us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 06:58:42.478950 (+ 229us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33214: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
W20260620 06:58:42.479296 9681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33214: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 06:58:42.479341 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9500: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45297: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 06:58:42.479478 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9500
I20260620 06:58:42.492157 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9150
I20260620 06:58:42.493494 9212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:42.604867 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9150
2026-06-20T06:58:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20634 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T06:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:42Z Disabled control of system clock
I20260620 06:58:42.672214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37047
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:58:42.980021 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:42.980281 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:42.980372 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:42.987272 9686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:42.987376 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:42.987412 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:42.987466 9686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:42.987509 9686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:42.996333 9686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:42.997553 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:42.999266 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:43.010051 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.010030 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.010208 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:43.011054 9686 server_base.cc:1034] running on GCE node
I20260620 06:58:43.011747 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:43.013024 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:43.014207 9686 hybrid_clock.cc:648] HybridClock initialized: now 1781938723014158 us; error 42 us; skew 500 ppm
I20260620 06:58:43.014514 9686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:43.016549 9686 webserver.cc:460] Webserver started at http://127.0.97.62:38033/ using document root <none> and password file <none>
I20260620 06:58:43.017181 9686 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:43.017277 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:43.017572 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:43.020366 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "8db644a7e777460d8d7278c1c805f515"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "53ab10cd266a5935288aab47cc176d0f"
server_key_iv: "e5d9ad2ce5e3048a0230a154152de79e"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.021080 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "8db644a7e777460d8d7278c1c805f515"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "53ab10cd266a5935288aab47cc176d0f"
server_key_iv: "e5d9ad2ce5e3048a0230a154152de79e"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.025924 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 06:58:43.029317 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.030719 9686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:58:43.030879 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "8db644a7e777460d8d7278c1c805f515"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "53ab10cd266a5935288aab47cc176d0f"
server_key_iv: "e5d9ad2ce5e3048a0230a154152de79e"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.031073 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:43.044584 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:43.045398 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:43.045680 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:43.062252 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37047
I20260620 06:58:43.062259 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37047 every 8 connection(s)
I20260620 06:58:43.063587 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:43.067471 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:43.070761 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9686
I20260620 06:58:43.070948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:43.071516 388 external_mini_cluster.cc:1428] Setting key 79813ae70c40731f02a0816de63d4725
I20260620 06:58:43.079953 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515: Bootstrap starting.
I20260620 06:58:43.083092 9752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:43.084183 9752 log.cc:824] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:43.085029 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:43.073498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.0.97.62:37047)
0620 06:58:43.074526 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:43.074567 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:43.074620 (+ 53us) server_negotiation.cc:407] Connection header received
0620 06:58:43.076546 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:43.076572 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:43.077107 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:43.077518 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:43.078104 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.079416 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.080729 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.081431 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.081598 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:43.082745 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:43.082752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:43.082766 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:43.083337 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:43.083441 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:43.083465 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:43.083578 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:43.083769 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 06:58:43.084069 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":383,"threads_started":1}
I20260620 06:58:43.086882 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515: No bootstrap required, opened a new log
I20260620 06:58:43.094058 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } }
I20260620 06:58:43.094383 9752 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:43.094471 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db644a7e777460d8d7278c1c805f515, State: Initialized, Role: FOLLOWER
I20260620 06:58:43.095065 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } }
I20260620 06:58:43.095283 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:43.095402 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:43.095544 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:43.097867 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } }
I20260620 06:58:43.098345 9752 leader_election.cc:302] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8db644a7e777460d8d7278c1c805f515; no voters:
I20260620 06:58:43.098889 9752 leader_election.cc:288] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:43.099025 9757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:43.099624 9757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [term 1 LEADER]: Becoming Leader. State: Replica: 8db644a7e777460d8d7278c1c805f515, State: Running, Role: LEADER
I20260620 06:58:43.100167 9757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } }
I20260620 06:58:43.100692 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:43.102739 9758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8db644a7e777460d8d7278c1c805f515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } } }
I20260620 06:58:43.103032 9758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:43.102818 9759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8db644a7e777460d8d7278c1c805f515. Latest consensus state: current_term: 1 leader_uuid: "8db644a7e777460d8d7278c1c805f515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8db644a7e777460d8d7278c1c805f515" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37047 } } }
I20260620 06:58:43.103178 9759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:43.103823 9766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:43.109552 9766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:43.120242 9766 catalog_manager.cc:1347] Generated new cluster ID: 4edd7d7941264f8b867896eb8359c76e
I20260620 06:58:43.120357 9766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:43.145488 9766 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:43.147372 9766 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:43.155920 9766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8db644a7e777460d8d7278c1c805f515: Generated new TSK 0
I20260620 06:58:43.156916 9766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:58:43.172932 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--builtin_ntp_servers=127.0.97.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 06:58:43.490737 9776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:43.491024 9776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:43.491118 9776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:43.498327 9776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:43.498436 9776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:43.498560 9776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:43.507740 9776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:43.509012 9776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:43.510810 9776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:43.523018 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.522989 9782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.523083 9784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:43.523847 9776 server_base.cc:1034] running on GCE node
I20260620 06:58:43.524405 9776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:43.525270 9776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:43.526458 9776 hybrid_clock.cc:648] HybridClock initialized: now 1781938723526405 us; error 46 us; skew 500 ppm
I20260620 06:58:43.526767 9776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:43.528805 9776 webserver.cc:460] Webserver started at http://127.0.97.1:40967/ using document root <none> and password file <none>
I20260620 06:58:43.529441 9776 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:43.529536 9776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:43.529829 9776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:43.532636 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "377e6dfcd082430d919382e9fba94d8d"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "d49dd07051785629972e1b467d59b9bf"
server_key_iv: "db0e2c76b89addb53adf832b5a65fbf9"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.533378 9776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "377e6dfcd082430d919382e9fba94d8d"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "d49dd07051785629972e1b467d59b9bf"
server_key_iv: "db0e2c76b89addb53adf832b5a65fbf9"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.538335 9776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 06:58:43.541606 9790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.542955 9776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:43.543097 9776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "377e6dfcd082430d919382e9fba94d8d"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "d49dd07051785629972e1b467d59b9bf"
server_key_iv: "db0e2c76b89addb53adf832b5a65fbf9"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.543283 9776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:43.559674 9776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:43.560480 9776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:43.560724 9776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:43.561582 9776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:43.563042 9776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:43.563133 9776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.563244 9776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:43.563310 9776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.592202 9776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46775
I20260620 06:58:43.592248 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46775 every 8 connection(s)
I20260620 06:58:43.593458 9776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:43.602946 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9776
I20260620 06:58:43.603134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:43.603785 388 external_mini_cluster.cc:1428] Setting key feb7fa5a7b527c03bd04316c57739395
I20260620 06:58:43.606113 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:43.595968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39809 (local address 127.0.97.62:37047)
0620 06:58:43.596423 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:43.596428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:43.597492 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 06:58:43.599249 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:43.599255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:43.599342 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:43.599434 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:43.600588 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.601350 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.602963 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.603475 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.603613 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:43.605102 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:43.605108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:43.605112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:43.605132 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:43.605195 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:43.605210 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:43.605296 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:43.605727 (+ 431us) server_negotiation.cc:299] Negotiation successful
0620 06:58:43.605831 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":269,"threads_started":1}
I20260620 06:58:43.606681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--builtin_ntp_servers=127.0.97.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:58:43.607195 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:43.596450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37047 (local address 127.0.97.1:39809)
0620 06:58:43.597331 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:43.597370 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:43.598988 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:43.599688 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:43.599697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:43.599796 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:43.600433 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:43.600453 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.601491 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.601496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:43.602833 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:43.602845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:43.603747 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:43.603754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:43.604062 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:43.604807 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:43.604810 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:43.604862 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:43.604942 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:43.605303 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:43.605311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:43.605317 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:58:43.605510 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 06:58:43.605898 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":282,"threads_started":1}
I20260620 06:58:43.608500 9903 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37047
I20260620 06:58:43.608805 9903 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:43.609541 9903 heartbeater.cc:511] Master 127.0.97.62:37047 requested a full tablet report, sending...
I20260620 06:58:43.611660 9717 ts_manager.cc:194] Registered new tserver with Master: 377e6dfcd082430d919382e9fba94d8d (127.0.97.1:46775)
I20260620 06:58:43.613296 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39809
W20260620 06:58:43.916267 9907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:43.916553 9907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:43.916675 9907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:43.923872 9907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:43.923977 9907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:43.924115 9907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:43.933055 9907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:43.934422 9907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:43.936256 9907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:43.948485 9912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.948463 9913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:43.948514 9915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:43.949452 9907 server_base.cc:1034] running on GCE node
I20260620 06:58:43.950045 9907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:43.951001 9907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:43.952196 9907 hybrid_clock.cc:648] HybridClock initialized: now 1781938723952150 us; error 36 us; skew 500 ppm
I20260620 06:58:43.952493 9907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:43.954612 9907 webserver.cc:460] Webserver started at http://127.0.97.2:45253/ using document root <none> and password file <none>
I20260620 06:58:43.955281 9907 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:43.955379 9907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:43.955673 9907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:43.958395 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4e2dc742623466c9d2c4f73ab084ba5"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "8f9a10c540916af5c330c4b2d53a7f1e"
server_key_iv: "29675386df684a00af74146430e996c0"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.959173 9907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4e2dc742623466c9d2c4f73ab084ba5"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "8f9a10c540916af5c330c4b2d53a7f1e"
server_key_iv: "29675386df684a00af74146430e996c0"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.964056 9907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:58:43.968053 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.969498 9907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:43.969643 9907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "f4e2dc742623466c9d2c4f73ab084ba5"
format_stamp: "Formatted at 2026-06-20 06:58:43 on dist-test-slave-mz20"
server_key: "8f9a10c540916af5c330c4b2d53a7f1e"
server_key_iv: "29675386df684a00af74146430e996c0"
server_key_version: "encryptionkey@0"
I20260620 06:58:43.969834 9907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:43.987612 9907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:43.988392 9907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:43.988636 9907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:43.989517 9907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:43.990933 9907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:43.991034 9907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:43.991134 9907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:43.991199 9907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:44.019388 9907 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33103
I20260620 06:58:44.019452 10033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33103 every 8 connection(s)
I20260620 06:58:44.020649 9907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:44.027424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 9907
I20260620 06:58:44.027606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:44.028079 388 external_mini_cluster.cc:1428] Setting key a5b03aef6abb40dfe91aee98ff105534
I20260620 06:58:44.030675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--builtin_ntp_servers=127.0.97.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:58:44.033174 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.023140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35029 (local address 127.0.97.62:37047)
0620 06:58:44.023343 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.023350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.024899 (+ 1549us) server_negotiation.cc:407] Connection header received
0620 06:58:44.026596 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.026602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.026675 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.026766 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.027887 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.028817 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.030468 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.031029 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.031138 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.032408 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:44.032414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:44.032418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:44.032440 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:44.032482 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:44.032494 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:44.032567 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:44.032769 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 06:58:44.032858 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:58:44.034188 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.023751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37047 (local address 127.0.97.2:35029)
0620 06:58:44.024745 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:44.024782 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:44.026284 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:44.026973 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:44.026982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:44.027074 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:44.027718 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:44.027735 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.028990 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.028995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:44.030339 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:44.030350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.031236 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.031243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:44.031465 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.032116 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:44.032118 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:44.032158 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:44.032239 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:44.032578 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:44.032589 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:44.032597 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:58:44.032746 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:58:44.033144 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":294,"threads_started":1}
I20260620 06:58:44.035580 10034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37047
I20260620 06:58:44.035868 10034 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:44.036672 10034 heartbeater.cc:511] Master 127.0.97.62:37047 requested a full tablet report, sending...
I20260620 06:58:44.038151 9717 ts_manager.cc:194] Registered new tserver with Master: f4e2dc742623466c9d2c4f73ab084ba5 (127.0.97.2:33103)
I20260620 06:58:44.038992 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35029
W20260620 06:58:44.351374 10037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:44.351671 10037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:44.351801 10037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:44.359323 10037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:44.359454 10037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:44.359612 10037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:44.368659 10037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:44.369925 10037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:44.371773 10037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:44.384157 10045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:44.384199 10043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:44.384224 10042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:44.384984 10037 server_base.cc:1034] running on GCE node
I20260620 06:58:44.385574 10037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:44.386512 10037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:44.387737 10037 hybrid_clock.cc:648] HybridClock initialized: now 1781938724387681 us; error 42 us; skew 500 ppm
I20260620 06:58:44.388044 10037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:44.390213 10037 webserver.cc:460] Webserver started at http://127.0.97.3:33335/ using document root <none> and password file <none>
I20260620 06:58:44.390924 10037 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:44.391017 10037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:44.391319 10037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:44.394086 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "64a0e6eecb3742bfa1750ba1956e7092"
format_stamp: "Formatted at 2026-06-20 06:58:44 on dist-test-slave-mz20"
server_key: "ead73c1b76a394f1b99cb3aed10dd615"
server_key_iv: "a16b3dbe751866c1a64148b1b389eb84"
server_key_version: "encryptionkey@0"
I20260620 06:58:44.394860 10037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "64a0e6eecb3742bfa1750ba1956e7092"
format_stamp: "Formatted at 2026-06-20 06:58:44 on dist-test-slave-mz20"
server_key: "ead73c1b76a394f1b99cb3aed10dd615"
server_key_iv: "a16b3dbe751866c1a64148b1b389eb84"
server_key_version: "encryptionkey@0"
I20260620 06:58:44.399842 10037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 06:58:44.403303 10051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:44.404624 10037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:44.404765 10037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "64a0e6eecb3742bfa1750ba1956e7092"
format_stamp: "Formatted at 2026-06-20 06:58:44 on dist-test-slave-mz20"
server_key: "ead73c1b76a394f1b99cb3aed10dd615"
server_key_iv: "a16b3dbe751866c1a64148b1b389eb84"
server_key_version: "encryptionkey@0"
I20260620 06:58:44.404958 10037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:44.421380 10037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:44.422297 10037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:44.422592 10037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:44.423527 10037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:44.424983 10037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:44.425065 10037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:44.425168 10037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:44.425232 10037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:44.452543 10037 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46397
I20260620 06:58:44.452595 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46397 every 8 connection(s)
I20260620 06:58:44.453769 10037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:44.462710 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10037
I20260620 06:58:44.462886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:44.463287 388 external_mini_cluster.cc:1428] Setting key c0fd16315c89bedb93b69984fb27fc3f
I20260620 06:58:44.467137 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.456593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42169 (local address 127.0.97.62:37047)
0620 06:58:44.456767 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.456772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.458240 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 06:58:44.460023 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.460028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.460097 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.460212 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.461510 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.462236 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.463986 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.464481 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.464593 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.466197 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:44.466203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:44.466207 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:44.466227 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:44.466265 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:44.466280 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:44.466377 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:44.466707 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 06:58:44.466786 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:58:44.468145 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.457110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37047 (local address 127.0.97.3:42169)
0620 06:58:44.458067 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:44.458116 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:44.459747 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:44.460545 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:44.460554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:44.460653 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:44.461263 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:44.461281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.462409 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.462412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:44.463845 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:44.463858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.464707 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.464715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:44.465000 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.465864 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:44.465868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:44.465919 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:44.466004 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:44.466418 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:44.466429 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:44.466439 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 06:58:44.466661 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 06:58:44.466981 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":345,"threads_started":1}
I20260620 06:58:44.469429 10164 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37047
I20260620 06:58:44.469710 10164 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:44.470467 10164 heartbeater.cc:511] Master 127.0.97.62:37047 requested a full tablet report, sending...
I20260620 06:58:44.471881 9717 ts_manager.cc:194] Registered new tserver with Master: 64a0e6eecb3742bfa1750ba1956e7092 (127.0.97.3:46397)
I20260620 06:58:44.472718 9717 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:42169
I20260620 06:58:44.478626 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:58:44.484216 9905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:44.482788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.97.62:37047)
0620 06:58:44.482968 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.482975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.483366 (+ 391us) server_negotiation.cc:407] Connection header received
0620 06:58:44.483491 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.483495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.483567 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.483644 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.484003 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48208: BlockingRecv error: recv got EOF from 127.0.0.1:48208 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:58:44.484560 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9776
I20260620 06:58:44.492046 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.485616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.0.97.1:46775)
0620 06:58:44.486076 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.486083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.486108 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:58:44.486162 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.486168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.486376 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.486494 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.486863 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.487825 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.489001 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.489539 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.489632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.490642 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:44.490652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:44.490666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:44.491260 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:44.491306 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:44.491324 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:44.491415 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:44.491643 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 06:58:44.491757 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":206,"threads_started":1}
I20260620 06:58:44.493482 9898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:44.573522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9776
I20260620 06:58:44.587181 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9907
I20260620 06:58:44.598098 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.588000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.97.2:33103)
0620 06:58:44.588521 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.588530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.588559 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:58:44.588616 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.588623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.588830 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.588965 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.589317 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.590380 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.591652 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.592231 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.592331 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.596718 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:44.596726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:44.596741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:44.597362 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:44.597414 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:44.597435 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:44.597527 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:44.597664 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 06:58:44.597790 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
I20260620 06:58:44.599565 10029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:44.680224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9907
I20260620 06:58:44.692857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10037
I20260620 06:58:44.700730 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:44.693714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.97.3:46397)
0620 06:58:44.694304 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:44.694311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:44.694342 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:58:44.694403 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:44.694418 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:44.694671 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:44.694808 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:44.695247 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.696336 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.697577 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:44.698132 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:44.698232 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:44.699299 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:44.699307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:44.699321 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:44.699895 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:44.699943 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:44.699962 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:44.700052 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:44.700202 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 06:58:44.700326 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
I20260620 06:58:44.702019 10159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:44.781252 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10037
I20260620 06:58:44.793294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 9686
I20260620 06:58:44.794445 9747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:44.891671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 9686
2026-06-20T06:58:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2282 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T06:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:44Z Disabled control of system clock
I20260620 06:58:44.953732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45991
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:58:45.266834 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:45.267148 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:45.267267 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:45.274504 10183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:45.274652 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:45.274717 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:45.274798 10183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:45.274855 10183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:45.284045 10183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:45.285290 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:45.287037 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:45.302801 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:45.302832 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:45.302865 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:45.304167 10183 server_base.cc:1034] running on GCE node
I20260620 06:58:45.304998 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:45.306490 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:45.307765 10183 hybrid_clock.cc:648] HybridClock initialized: now 1781938725307713 us; error 48 us; skew 500 ppm
I20260620 06:58:45.308137 10183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:45.310434 10183 webserver.cc:460] Webserver started at http://127.0.97.62:34771/ using document root <none> and password file <none>
I20260620 06:58:45.311160 10183 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:45.311259 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:45.311561 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:45.314389 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "795e7f0643ef6318ae5694ac2a2e32e0"
server_key_iv: "c80a19d95163c56c1ba22e7537f99b3d"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.315200 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "795e7f0643ef6318ae5694ac2a2e32e0"
server_key_iv: "c80a19d95163c56c1ba22e7537f99b3d"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.320554 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 06:58:45.324359 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:45.325928 10183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:58:45.326118 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "795e7f0643ef6318ae5694ac2a2e32e0"
server_key_iv: "c80a19d95163c56c1ba22e7537f99b3d"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.326340 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:45.332167 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:45.332928 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:45.333196 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:45.351011 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45991
I20260620 06:58:45.351054 10248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45991 every 8 connection(s)
I20260620 06:58:45.352288 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:45.353679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10183
I20260620 06:58:45.353858 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:45.354664 388 external_mini_cluster.cc:1428] Setting key 5374552c69c54932847cbe86000418ca
I20260620 06:58:45.356907 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:45.362757 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:45.356636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.0.97.62:45991)
0620 06:58:45.357580 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:45.357597 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:45.357640 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:58:45.359231 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:45.359252 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:45.359685 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:45.360011 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:45.361117 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:45.361124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:45.361137 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:45.361152 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:45.361235 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:45.361252 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:45.361432 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:45.361536 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 06:58:45.361790 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":334,"threads_started":1}
I20260620 06:58:45.368433 10249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153: Bootstrap starting.
I20260620 06:58:45.371385 10249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:45.372344 10249 log.cc:824] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:45.374960 10249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153: No bootstrap required, opened a new log
I20260620 06:58:45.381827 10249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } }
I20260620 06:58:45.382159 10249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:45.382251 10249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a6511f8c7c6454f9d3b4e02ddfcf153, State: Initialized, Role: FOLLOWER
I20260620 06:58:45.382818 10249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } }
I20260620 06:58:45.383014 10249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:45.383141 10249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:45.383284 10249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:45.385277 10249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } }
I20260620 06:58:45.385739 10249 leader_election.cc:302] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a6511f8c7c6454f9d3b4e02ddfcf153; no voters:
I20260620 06:58:45.386281 10249 leader_election.cc:288] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:45.386413 10254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:45.387411 10254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [term 1 LEADER]: Becoming Leader. State: Replica: 8a6511f8c7c6454f9d3b4e02ddfcf153, State: Running, Role: LEADER
I20260620 06:58:45.387996 10254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } }
I20260620 06:58:45.388362 10249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:45.390739 10256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a6511f8c7c6454f9d3b4e02ddfcf153. Latest consensus state: current_term: 1 leader_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } } }
I20260620 06:58:45.391181 10256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:45.391074 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a6511f8c7c6454f9d3b4e02ddfcf153" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45991 } } }
I20260620 06:58:45.391348 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:45.392768 10261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:45.397519 10261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:45.406122 10261 catalog_manager.cc:1347] Generated new cluster ID: 32ba445b20db4b5ab55fc83cc0eb05dc
I20260620 06:58:45.406229 10261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:45.425784 10261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:45.426301 10261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:45.433038 10261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a6511f8c7c6454f9d3b4e02ddfcf153: Generated new TSK 0
I20260620 06:58:45.433842 10261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:58:45.448112 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--builtin_ntp_servers=127.0.97.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:58:45.768579 10273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:45.768857 10273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:45.769007 10273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:45.776185 10273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:45.776299 10273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:45.776430 10273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:45.785272 10273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:45.786499 10273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:45.788232 10273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:45.800805 10278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:45.800869 10279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:45.800837 10281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:45.801826 10273 server_base.cc:1034] running on GCE node
I20260620 06:58:45.802397 10273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:45.803313 10273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:45.804519 10273 hybrid_clock.cc:648] HybridClock initialized: now 1781938725804454 us; error 55 us; skew 500 ppm
I20260620 06:58:45.804847 10273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:45.807049 10273 webserver.cc:460] Webserver started at http://127.0.97.1:43193/ using document root <none> and password file <none>
I20260620 06:58:45.807708 10273 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:45.807801 10273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:45.808079 10273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:45.812162 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "d31e7a9ecace49a6a808d3817da350c4"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "e43377d3f6633dd68c6f8b0d6782fe4b"
server_key_iv: "8f2a6f2dc2173323f1b068509942b235"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.812897 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d31e7a9ecace49a6a808d3817da350c4"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "e43377d3f6633dd68c6f8b0d6782fe4b"
server_key_iv: "8f2a6f2dc2173323f1b068509942b235"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.817741 10273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 06:58:45.821090 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:45.822432 10273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:58:45.822625 10273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d31e7a9ecace49a6a808d3817da350c4"
format_stamp: "Formatted at 2026-06-20 06:58:45 on dist-test-slave-mz20"
server_key: "e43377d3f6633dd68c6f8b0d6782fe4b"
server_key_iv: "8f2a6f2dc2173323f1b068509942b235"
server_key_version: "encryptionkey@0"
I20260620 06:58:45.822836 10273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:45.828595 10273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:45.829319 10273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:45.829540 10273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:45.830430 10273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:45.831799 10273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:45.831892 10273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:45.831988 10273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:45.832053 10273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:45.860494 10273 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37045
I20260620 06:58:45.860534 10399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37045 every 8 connection(s)
I20260620 06:58:45.861766 10273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:45.869164 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:45.864174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48175 (local address 127.0.97.62:45991)
0620 06:58:45.864551 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:45.864556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:45.865746 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 06:58:45.867345 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:45.867349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:45.867398 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:45.867489 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:45.868584 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:45.868590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:45.868594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:45.868611 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:45.868650 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:45.868662 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:45.868749 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:45.868877 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 06:58:45.868924 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":193,"threads_started":1}
I20260620 06:58:45.870002 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:45.864743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45991 (local address 127.0.97.1:48175)
0620 06:58:45.865590 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:45.865631 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:45.867104 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:45.867662 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:45.867670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:45.867756 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:45.868326 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:45.868328 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:45.868363 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:45.868436 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:45.868731 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:45.868737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:45.868741 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:58:45.868927 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 06:58:45.869109 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":246,"threads_started":1}
I20260620 06:58:45.871238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10273
I20260620 06:58:45.871397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:45.871524 10400 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45991
I20260620 06:58:45.871822 10400 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:45.872141 388 external_mini_cluster.cc:1428] Setting key ce195df9dc4917fca645a1274da8d461
I20260620 06:58:45.872494 10400 heartbeater.cc:511] Master 127.0.97.62:45991 requested a full tablet report, sending...
I20260620 06:58:45.874516 10213 ts_manager.cc:194] Registered new tserver with Master: d31e7a9ecace49a6a808d3817da350c4 (127.0.97.1:37045)
I20260620 06:58:45.874814 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--builtin_ntp_servers=127.0.97.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:58:46.187721 10404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:46.188009 10404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:46.188095 10404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:46.195593 10404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:46.195712 10404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:46.195822 10404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:46.204948 10404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:46.206223 10404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:46.207959 10404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:46.220294 10409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:46.220358 10410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:46.220321 10412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:46.221421 10404 server_base.cc:1034] running on GCE node
I20260620 06:58:46.221964 10404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:46.222869 10404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:46.224061 10404 hybrid_clock.cc:648] HybridClock initialized: now 1781938726224011 us; error 39 us; skew 500 ppm
I20260620 06:58:46.224354 10404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:46.226490 10404 webserver.cc:460] Webserver started at http://127.0.97.2:40659/ using document root <none> and password file <none>
I20260620 06:58:46.227219 10404 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:46.227311 10404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:46.227592 10404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:46.230369 10404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "7fb96dbc60354d17b32e4b0d336b25fe"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "914b91676d91db00f5d3ee76a1be6a53"
server_key_iv: "68d3329ef4363d0750a0739ba20880f0"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.231143 10404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7fb96dbc60354d17b32e4b0d336b25fe"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "914b91676d91db00f5d3ee76a1be6a53"
server_key_iv: "68d3329ef4363d0750a0739ba20880f0"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.236003 10404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:58:46.239209 10418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.240485 10404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:58:46.240617 10404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "7fb96dbc60354d17b32e4b0d336b25fe"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "914b91676d91db00f5d3ee76a1be6a53"
server_key_iv: "68d3329ef4363d0750a0739ba20880f0"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.240826 10404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:46.246161 10404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:46.246961 10404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:46.247218 10404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:46.248046 10404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:46.249377 10404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:46.249485 10404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.249584 10404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:46.249657 10404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.279210 10404 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36817
I20260620 06:58:46.279284 10530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36817 every 8 connection(s)
I20260620 06:58:46.280454 10404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:46.284598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10404
I20260620 06:58:46.284737 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:46.285141 388 external_mini_cluster.cc:1428] Setting key bb61bb4d47bbf12adff9c45c8b944079
I20260620 06:58:46.288106 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--builtin_ntp_servers=127.0.97.20:34723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 06:58:46.288453 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.282896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56271 (local address 127.0.97.62:45991)
0620 06:58:46.283108 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.283115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.284459 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 06:58:46.286237 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.286242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.286296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.286397 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.287760 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:46.287766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:46.287771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:46.287792 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:46.287831 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:46.287845 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:46.287961 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:46.288121 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 06:58:46.288193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:58:46.289423 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.283417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45991 (local address 127.0.97.2:56271)
0620 06:58:46.284321 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:46.284357 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:46.285980 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:46.286654 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:46.286663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:46.286763 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:46.287463 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:46.287466 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:46.287504 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:46.287590 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:46.287931 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:46.287938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:46.287944 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:58:46.288155 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:58:46.288428 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":258,"threads_started":1}
I20260620 06:58:46.290884 10531 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45991
I20260620 06:58:46.291148 10531 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:46.291707 10531 heartbeater.cc:511] Master 127.0.97.62:45991 requested a full tablet report, sending...
I20260620 06:58:46.293037 10213 ts_manager.cc:194] Registered new tserver with Master: 7fb96dbc60354d17b32e4b0d336b25fe (127.0.97.2:36817)
W20260620 06:58:46.597584 10534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:46.597859 10534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:46.597944 10534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:46.605195 10534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:46.605300 10534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:46.605414 10534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:46.614532 10534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:46.615765 10534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:46.617470 10534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:46.630282 10542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:46.630391 10540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:46.630415 10539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:46.631464 10534 server_base.cc:1034] running on GCE node
I20260620 06:58:46.632031 10534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:46.633021 10534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:46.634238 10534 hybrid_clock.cc:648] HybridClock initialized: now 1781938726634182 us; error 48 us; skew 500 ppm
I20260620 06:58:46.634569 10534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 06:58:46.636734 10534 webserver.cc:460] Webserver started at http://127.0.97.3:36791/ using document root <none> and password file <none>
I20260620 06:58:46.637426 10534 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:46.637521 10534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:46.637822 10534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:46.640574 10534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "d27465fd722b4a9082749829e9f0c367"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "2215cf3c751cc95c5fd9c5084aac6dd9"
server_key_iv: "e0e10e5160f640cc488e4673728fecf2"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.641300 10534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d27465fd722b4a9082749829e9f0c367"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "2215cf3c751cc95c5fd9c5084aac6dd9"
server_key_iv: "e0e10e5160f640cc488e4673728fecf2"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.646217 10534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 06:58:46.649665 10548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.651108 10534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:58:46.651280 10534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "d27465fd722b4a9082749829e9f0c367"
format_stamp: "Formatted at 2026-06-20 06:58:46 on dist-test-slave-mz20"
server_key: "2215cf3c751cc95c5fd9c5084aac6dd9"
server_key_iv: "e0e10e5160f640cc488e4673728fecf2"
server_key_version: "encryptionkey@0"
I20260620 06:58:46.651487 10534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:46.656941 10534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:46.657739 10534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:46.657970 10534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:46.658905 10534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:46.660223 10534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:46.660315 10534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.660418 10534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:46.660485 10534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:46.689630 10534 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39413
I20260620 06:58:46.689687 10660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39413 every 8 connection(s)
I20260620 06:58:46.690960 10534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:46.698396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10534
I20260620 06:58:46.698628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:46.699124 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.693368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47507 (local address 127.0.97.62:45991)
0620 06:58:46.693553 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.693558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.695492 (+ 1934us) server_negotiation.cc:407] Connection header received
0620 06:58:46.697088 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.697092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.697144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.697223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.698372 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:46.698378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:46.698382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:46.698402 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:46.698440 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:46.698454 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:46.698627 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:46.698794 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 06:58:46.698860 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:58:46.699164 388 external_mini_cluster.cc:1428] Setting key 083fe5165f36e37675f3ef22608647f3
I20260620 06:58:46.700076 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.694016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45991 (local address 127.0.97.3:47507)
0620 06:58:46.695332 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:46.695377 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:46.696831 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:46.697417 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:46.697425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:46.697518 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:46.698085 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:58:46.698087 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:46.698123 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:58:46.698204 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:46.698563 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:46.698574 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:46.698580 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:58:46.698835 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 06:58:46.699034 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":573,"threads_started":1}
I20260620 06:58:46.701344 10661 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45991
I20260620 06:58:46.701596 10661 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:46.702150 10661 heartbeater.cc:511] Master 127.0.97.62:45991 requested a full tablet report, sending...
I20260620 06:58:46.703387 10213 ts_manager.cc:194] Registered new tserver with Master: d27465fd722b4a9082749829e9f0c367 (127.0.97.3:39413)
I20260620 06:58:46.713943 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:58:46.719604 10402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:58:46.718114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.97.62:45991)
0620 06:58:46.718318 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.718325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.718707 (+ 382us) server_negotiation.cc:407] Connection header received
0620 06:58:46.718854 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.718858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.718914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.719085 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.719371 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34206: BlockingRecv error: recv got EOF from 127.0.0.1:34206 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:58:46.720140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10273
I20260620 06:58:46.723269 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.720987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.97.1:37045)
0620 06:58:46.721445 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.721451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.721475 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:58:46.721525 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.721531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.721723 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.721843 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.722675 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:46.722681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:46.722692 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:46.722703 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:46.722746 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:46.722758 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:46.722861 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:46.722927 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 06:58:46.723009 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":218,"threads_started":1}
I20260620 06:58:46.724419 10395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:46.804068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10273
I20260620 06:58:46.816953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10404
I20260620 06:58:46.820406 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.817825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.97.2:36817)
0620 06:58:46.818304 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.818311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.818337 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:58:46.818399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.818406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.818669 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.818918 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.819774 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:46.819782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:46.819794 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:46.819811 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:46.819867 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:46.819884 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:46.820003 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:46.820095 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 06:58:46.820179 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":208,"threads_started":1}
I20260620 06:58:46.821641 10526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:46.900354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10404
I20260620 06:58:46.912676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10534
I20260620 06:58:46.916200 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:46.913511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.0.97.3:39413)
0620 06:58:46.914075 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:46.914082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:46.914111 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:58:46.914170 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:46.914175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:46.914395 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:46.914514 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:46.915531 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:46.915537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:46.915549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:58:46.915559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:46.915607 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:58:46.915620 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:46.915750 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:46.915797 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:58:46.915879 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":274,"threads_started":1}
I20260620 06:58:46.917328 10656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:46.996976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10534
I20260620 06:58:47.009485 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10183
I20260620 06:58:47.010628 10244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:47.108467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10183
2026-06-20T06:58:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2217 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:47 dist-test-slave-mz20 krb5kdc[10679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:47 dist-test-slave-mz20 krb5kdc[10679](info): set up 2 sockets
Jun 20 06:58:47 dist-test-slave-mz20 krb5kdc[10679](info): commencing operation
krb5kdc: starting...
W20260620 06:58:48.111159 388 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:48 dist-test-slave-mz20 krb5kdc[10679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:58:48.321358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32869
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:48.625366 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:48.625659 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:48.625774 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:48.632872 10692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:58:48.632988 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:48.633036 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:48.633111 10692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:58:48.633164 10692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:58:48.642927 10692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:48.644243 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:48.645524 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:48.656545 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:48.656505 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:48.657004 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:48.657362 10692 server_base.cc:1034] running on GCE node
I20260620 06:58:48.658181 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:48.659614 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:48.660818 10692 hybrid_clock.cc:648] HybridClock initialized: now 1781938728660760 us; error 48 us; skew 500 ppm
I20260620 06:58:48.661171 10692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:48 dist-test-slave-mz20 krb5kdc[10679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:48.664173 10692 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:58:48.665519 10692 webserver.cc:460] Webserver started at http://127.0.97.62:45709/ using document root <none> and password file <none>
I20260620 06:58:48.666160 10692 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:48.666270 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:48.666568 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:48.669384 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "c86e9e2b405c489a9edb95b47d488035"
format_stamp: "Formatted at 2026-06-20 06:58:48 on dist-test-slave-mz20"
server_key: "cf627707d3dd57aeb30f6dc9fc166c47"
server_key_iv: "a7545c284b13fd958dac658abcf12fca"
server_key_version: "encryptionkey@0"
I20260620 06:58:48.670146 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "c86e9e2b405c489a9edb95b47d488035"
format_stamp: "Formatted at 2026-06-20 06:58:48 on dist-test-slave-mz20"
server_key: "cf627707d3dd57aeb30f6dc9fc166c47"
server_key_iv: "a7545c284b13fd958dac658abcf12fca"
server_key_version: "encryptionkey@0"
I20260620 06:58:48.675333 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 06:58:48.678776 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:48.680145 10692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:58:48.680305 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "c86e9e2b405c489a9edb95b47d488035"
format_stamp: "Formatted at 2026-06-20 06:58:48 on dist-test-slave-mz20"
server_key: "cf627707d3dd57aeb30f6dc9fc166c47"
server_key_iv: "a7545c284b13fd958dac658abcf12fca"
server_key_version: "encryptionkey@0"
I20260620 06:58:48.680480 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:48.695834 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:48.699878 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:48.700148 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:48.717286 10692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32869
I20260620 06:58:48.717303 10758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32869 every 8 connection(s)
I20260620 06:58:48.718588 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:58:48.721138 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10692
I20260620 06:58:48.721335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:58:48.722052 388 external_mini_cluster.cc:1428] Setting key e5485d2df9f77d84992547e3d63c466d
I20260620 06:58:48.722817 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:48.734457 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: Bootstrap starting.
Jun 20 06:58:48 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:48.737661 10759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:48.739001 10759 log.cc:824] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:48.741732 10759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: No bootstrap required, opened a new log
I20260620 06:58:48.743767 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:48.724064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.0.97.62:32869)
0620 06:58:48.724757 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:48.724769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:48.724809 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:58:48.727204 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:48.727226 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:48.727617 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:48.727983 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:48.729345 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:48.730607 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:48.731835 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:48.732569 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:48.732739 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:48.736728 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:48.736736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:48.736750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:48.736785 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:48.740078 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:48.740721 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:48.740728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:48.740735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:48.740891 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:48.741233 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:48.741237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:48.741239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:48.741715 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:48.741975 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:48.742384 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 06:58:48.742795 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
I20260620 06:58:48.749461 10759 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } }
I20260620 06:58:48.749866 10759 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:48.749974 10759 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86e9e2b405c489a9edb95b47d488035, State: Initialized, Role: FOLLOWER
I20260620 06:58:48.750523 10759 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } }
I20260620 06:58:48.750768 10759 raft_consensus.cc:388] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:58:48.750887 10759 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:58:48.751051 10759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:48.753099 10759 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } }
I20260620 06:58:48.753558 10759 leader_election.cc:302] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c86e9e2b405c489a9edb95b47d488035; no voters:
I20260620 06:58:48.754066 10759 leader_election.cc:288] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:58:48.754256 10764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:48.754878 10764 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [term 1 LEADER]: Becoming Leader. State: Replica: c86e9e2b405c489a9edb95b47d488035, State: Running, Role: LEADER
I20260620 06:58:48.755344 10764 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } }
I20260620 06:58:48.756047 10759 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:58:48.757899 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86e9e2b405c489a9edb95b47d488035. Latest consensus state: current_term: 1 leader_uuid: "c86e9e2b405c489a9edb95b47d488035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } } }
I20260620 06:58:48.757912 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c86e9e2b405c489a9edb95b47d488035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e9e2b405c489a9edb95b47d488035" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32869 } } }
I20260620 06:58:48.758285 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:48.758282 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035 [sys.catalog]: This master's current role is: LEADER
I20260620 06:58:48.759231 10770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:58:48.765779 10770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:58:48.774407 10770 catalog_manager.cc:1347] Generated new cluster ID: 78fc9e6ccba348ffbdd3b35d5f13e36e
I20260620 06:58:48.774529 10770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:58:48.789755 10770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:58:48.791136 10770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:58:48.803179 10770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: Generated new TSK 0
I20260620 06:58:48.803992 10770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:58:48.854291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:49.189775 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:49.190040 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:49.190129 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:49.197319 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:49.197422 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:49.197538 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:58:49.207317 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:49.208523 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:49.209741 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:49.222141 10790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:49.222117 10793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:49.222090 10791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:49.222951 10785 server_base.cc:1034] running on GCE node
I20260620 06:58:49.223568 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:49.224503 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:49.225698 10785 hybrid_clock.cc:648] HybridClock initialized: now 1781938729225640 us; error 52 us; skew 500 ppm
I20260620 06:58:49.225998 10785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:49 dist-test-slave-mz20 krb5kdc[10679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:49.228929 10785 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:58:49.230261 10785 webserver.cc:460] Webserver started at http://127.0.97.1:32977/ using document root <none> and password file <none>
I20260620 06:58:49.230953 10785 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:49.231060 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:49.231333 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:49.234144 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "25a148d1be2f4e149286508ecd3f507e"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "474930c2310a2f6d5ddc0714d18869c5"
server_key_iv: "bd2812d7ca83f9e7fafdcec4d170f6e9"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.234870 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "25a148d1be2f4e149286508ecd3f507e"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "474930c2310a2f6d5ddc0714d18869c5"
server_key_iv: "bd2812d7ca83f9e7fafdcec4d170f6e9"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.239843 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:58:49.243012 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.244347 10785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:58:49.244467 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "25a148d1be2f4e149286508ecd3f507e"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "474930c2310a2f6d5ddc0714d18869c5"
server_key_iv: "bd2812d7ca83f9e7fafdcec4d170f6e9"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.244614 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:49.270843 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:49.274896 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:49.275156 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:49.276021 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:49.277349 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:49.277423 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.277511 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:49.277554 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.307016 10785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39931
I20260620 06:58:49.307062 10912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39931 every 8 connection(s)
I20260620 06:58:49.308207 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:58:49.317832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10785
I20260620 06:58:49.318023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:58:49.318713 388 external_mini_cluster.cc:1428] Setting key 6d631ae81b20054777f62d3efba243ef
Jun 20 06:58:49 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:49.327652 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:49.310601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57823 (local address 127.0.97.62:32869)
0620 06:58:49.310985 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:49.310990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:49.311845 (+ 855us) server_negotiation.cc:407] Connection header received
0620 06:58:49.313586 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:49.313600 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:49.313674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:49.313763 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:49.315240 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.315933 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.317430 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.317820 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.317933 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:49.321723 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:49.321729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:49.321735 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:49.321776 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:49.324712 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:49.325422 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:49.325427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:49.325430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:49.325557 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:49.325920 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:49.325924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:49.325927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:49.326254 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:49.326381 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:49.327100 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 06:58:49.327305 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":204,"threads_started":1}
I20260620 06:58:49.328590 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:49.311052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32869 (local address 127.0.97.1:57823)
0620 06:58:49.311701 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:49.311738 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:49.313330 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:49.313990 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:49.313998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:49.314490 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:49.315077 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:49.315104 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.316085 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.316094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:49.317292 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:49.317303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.318031 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.318037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:49.318303 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:49.318978 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:49.319009 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:49.321482 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:49.324852 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:49.324859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:49.324873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:49.325305 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:49.325650 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:49.325653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:49.325655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:49.325830 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:49.326363 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:49.326368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:49.326737 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 06:58:49.326964 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 06:58:49.327353 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":248,"threads_started":1}
I20260620 06:58:49.330241 10913 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32869
I20260620 06:58:49.330531 10913 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:49.331310 10913 heartbeater.cc:511] Master 127.0.97.62:32869 requested a full tablet report, sending...
I20260620 06:58:49.333516 10724 ts_manager.cc:194] Registered new tserver with Master: 25a148d1be2f4e149286508ecd3f507e (127.0.97.1:39931)
I20260620 06:58:49.335393 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57823
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:58:49.367966 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:49.678383 10919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:49.678705 10919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:49.678812 10919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:49.685891 10919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:49.685990 10919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:49.686139 10919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:58:49.695485 10919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:49.696679 10919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:49.697930 10919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:49.710608 10924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:49.710637 10927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:49.710608 10925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:49.711344 10919 server_base.cc:1034] running on GCE node
I20260620 06:58:49.711956 10919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:49.712837 10919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:49.714037 10919 hybrid_clock.cc:648] HybridClock initialized: now 1781938729713987 us; error 48 us; skew 500 ppm
I20260620 06:58:49.714329 10919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:49 dist-test-slave-mz20 krb5kdc[10679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:49.721877 10919 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:58:49.723580 10919 webserver.cc:460] Webserver started at http://127.0.97.2:35423/ using document root <none> and password file <none>
I20260620 06:58:49.724400 10919 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:49.724512 10919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:49.724799 10919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:49.728456 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "356ec81ccc4e454c83037e1c8dfb7744"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "0689091dd335fa53e479202378b96f18"
server_key_iv: "eabb6248121df72c6ca5497122e5e159"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.729368 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "356ec81ccc4e454c83037e1c8dfb7744"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "0689091dd335fa53e479202378b96f18"
server_key_iv: "eabb6248121df72c6ca5497122e5e159"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.735504 10919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 06:58:49.738727 10934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.740007 10919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:58:49.740151 10919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "356ec81ccc4e454c83037e1c8dfb7744"
format_stamp: "Formatted at 2026-06-20 06:58:49 on dist-test-slave-mz20"
server_key: "0689091dd335fa53e479202378b96f18"
server_key_iv: "eabb6248121df72c6ca5497122e5e159"
server_key_version: "encryptionkey@0"
I20260620 06:58:49.740327 10919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:49.754133 10919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:49.758262 10919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:49.758507 10919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:49.759450 10919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:49.760741 10919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:49.760807 10919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.760910 10919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:49.760960 10919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:49.789937 10919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40209
I20260620 06:58:49.789973 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40209 every 8 connection(s)
I20260620 06:58:49.791172 10919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:58:49.797829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 10919
I20260620 06:58:49.798002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:58:49.798517 388 external_mini_cluster.cc:1428] Setting key 2ca32337f91fd079ce530a0952934532
Jun 20 06:58:49 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:49.809731 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:49.793594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58463 (local address 127.0.97.62:32869)
0620 06:58:49.793747 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:49.793752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:49.794882 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 06:58:49.796612 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:49.796618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:49.796700 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:49.796784 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:49.798570 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.799335 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.800822 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.801212 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.801311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:49.804689 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:49.804694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:49.804697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:49.804728 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:49.806971 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:49.807631 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:49.807636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:49.807638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:49.807765 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:49.808136 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:49.808141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:49.808144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:49.808402 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:49.808521 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:49.809177 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 06:58:49.809400 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:58:49.810712 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:49.794045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32869 (local address 127.0.97.2:58463)
0620 06:58:49.794732 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:49.794774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:49.796356 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:49.797016 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:49.797024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:49.797572 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:49.798365 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:49.798391 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.799482 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.799486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:49.800692 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:49.800704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:49.801407 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:49.801414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:49.801613 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:49.802245 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:49.802265 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:49.804484 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:49.807108 (+ 2624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:49.807116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:49.807133 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:49.807526 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:49.807858 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:49.807863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:49.807867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:49.808041 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:49.808514 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:49.808521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:49.808843 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 06:58:49.809080 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 06:58:49.809624 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20260620 06:58:49.811992 11047 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32869
I20260620 06:58:49.812297 11047 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:49.813063 11047 heartbeater.cc:511] Master 127.0.97.62:32869 requested a full tablet report, sending...
I20260620 06:58:49.814600 10724 ts_manager.cc:194] Registered new tserver with Master: 356ec81ccc4e454c83037e1c8dfb7744 (127.0.97.2:40209)
I20260620 06:58:49.815671 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58463
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:58:49.844897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:58:50.160651 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:58:50.160931 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:58:50.161060 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:58:50.168473 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:58:50.168579 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:58:50.168742 11052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:58:50.178424 11052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:58:50.179738 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:58:50.181075 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:58:50.193590 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:50.193516 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:58:50.193553 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:58:50.194185 11052 server_base.cc:1034] running on GCE node
I20260620 06:58:50.194815 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:58:50.195670 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:58:50.196873 11052 hybrid_clock.cc:648] HybridClock initialized: now 1781938730196823 us; error 45 us; skew 500 ppm
I20260620 06:58:50.197172 11052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:58:50 dist-test-slave-mz20 krb5kdc[10679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:58:50.200172 11052 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:58:50.201452 11052 webserver.cc:460] Webserver started at http://127.0.97.3:44479/ using document root <none> and password file <none>
I20260620 06:58:50.202092 11052 fs_manager.cc:359] Metadata directory not provided
I20260620 06:58:50.202181 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:58:50.202409 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:58:50.205179 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "76db6c2f674f400d8662da9301a95437"
format_stamp: "Formatted at 2026-06-20 06:58:50 on dist-test-slave-mz20"
server_key: "540af78f40e86bcb2949a00f5851936c"
server_key_iv: "9c8f88d30826bc311a95edcdcb11550e"
server_key_version: "encryptionkey@0"
I20260620 06:58:50.205868 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "76db6c2f674f400d8662da9301a95437"
format_stamp: "Formatted at 2026-06-20 06:58:50 on dist-test-slave-mz20"
server_key: "540af78f40e86bcb2949a00f5851936c"
server_key_iv: "9c8f88d30826bc311a95edcdcb11550e"
server_key_version: "encryptionkey@0"
I20260620 06:58:50.210604 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 06:58:50.213829 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:50.215291 11052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:58:50.215430 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "76db6c2f674f400d8662da9301a95437"
format_stamp: "Formatted at 2026-06-20 06:58:50 on dist-test-slave-mz20"
server_key: "540af78f40e86bcb2949a00f5851936c"
server_key_iv: "9c8f88d30826bc311a95edcdcb11550e"
server_key_version: "encryptionkey@0"
I20260620 06:58:50.215611 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:58:50.236944 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:58:50.241058 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:58:50.241308 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:58:50.242203 11052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:58:50.243567 11052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:58:50.243649 11052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:50.243753 11052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:58:50.243798 11052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:58:50.272902 11052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44803
I20260620 06:58:50.272950 11179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44803 every 8 connection(s)
I20260620 06:58:50.274132 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:58:50.275414 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 11052
I20260620 06:58:50.275573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:58:50.275981 388 external_mini_cluster.cc:1428] Setting key 7e20dda56ac241e103638a25727bb946
Jun 20 06:58:50 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:58:50.293246 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.276602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59137 (local address 127.0.97.62:32869)
0620 06:58:50.276797 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.276803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.278226 (+ 1423us) server_negotiation.cc:407] Connection header received
0620 06:58:50.280117 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.280121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.280182 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.280260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:50.281616 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.282342 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.283848 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.284238 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.284336 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.287635 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:50.287640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:50.287643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:50.287676 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:50.290414 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.291284 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.291288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.291290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.291410 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.291779 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.291782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.291784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.292032 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:50.292135 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.292853 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.293007 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:58:50.294193 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.277189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32869 (local address 127.0.97.3:59137)
0620 06:58:50.278014 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.278060 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.279897 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.280438 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.280445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.280928 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:58:50.281499 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.281516 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.282465 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.282468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.283727 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.283739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.284427 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.284431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.284637 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.285309 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:58:50.285333 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:58:50.287418 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:58:50.290586 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:50.290595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:50.290612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:50.291148 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:50.291491 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:58:50.291494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:58:50.291497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:58:50.291675 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:58:50.292118 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:58:50.292123 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:58:50.292468 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.292734 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.293123 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":318,"threads_started":1}
I20260620 06:58:50.295495 11180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32869
I20260620 06:58:50.295805 11180 heartbeater.cc:464] Registering TS with master...
I20260620 06:58:50.296490 11180 heartbeater.cc:511] Master 127.0.97.62:32869 requested a full tablet report, sending...
I20260620 06:58:50.297848 10724 ts_manager.cc:194] Registered new tserver with Master: 76db6c2f674f400d8662da9301a95437 (127.0.97.3:44803)
I20260620 06:58:50.298642 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59137
I20260620 06:58:50.303052 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:58:50.319205 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.307346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.0.97.62:32869)
0620 06:58:50.307518 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.307524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.307810 (+ 286us) server_negotiation.cc:407] Connection header received
0620 06:58:50.307929 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.307934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.308015 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.308115 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:50.309300 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.310371 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.311649 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.312058 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.312149 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.314177 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:50.314182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:50.314185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:50.314216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:50.316621 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.317234 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.317239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.317241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.317370 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.317714 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.317719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.317721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.318199 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:50.318406 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.318769 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.318927 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:58:50.323331 10724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:58:50.329937 10724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:58:50.338323 10913 heartbeater.cc:503] Master 127.0.97.62:32869 was elected leader, sending a full tablet report...
I20260620 06:58:50.352674 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.341457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44803 (local address 127.0.0.1:45436)
0620 06:58:50.342128 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.342172 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.342338 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.343736 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.343745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.343778 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:50.344378 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.344390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.347261 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.347268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.349840 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.349853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.351616 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.351622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.351846 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.351893 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.352013 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.352202 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":2,"thread_start_us":603,"threads_started":1}
I20260620 06:58:50.353363 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.341788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.97.3:44803)
0620 06:58:50.343064 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.343073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.343106 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:58:50.343207 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.343216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.343481 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.344583 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:50.344870 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.347121 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.350007 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.351430 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.351552 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.352833 (+ 1281us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.352906 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.353064 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":1873,"threads_started":1}
I20260620 06:58:50.354921 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.341607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39931 (local address 127.0.0.1:39056)
0620 06:58:50.345486 (+ 3879us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.345504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.345615 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.347712 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.347717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.347741 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:50.348170 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.348182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.350004 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.350008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.351431 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.351442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.353376 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.353380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.354290 (+ 910us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.354320 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.354409 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.354570 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3765,"mutex_wait_us":6,"thread_start_us":3636,"threads_started":1}
I20260620 06:58:50.354944 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.341857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.0.97.1:39931)
0620 06:58:50.346988 (+ 5131us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.346996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.347021 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:58:50.347117 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.347124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.347379 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.347557 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:50.348377 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.349873 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.351609 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.352949 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.353060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.353267 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.354454 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.354632 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4978,"thread_start_us":4853,"threads_started":1}
I20260620 06:58:50.354987 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.343507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.97.2:40209)
0620 06:58:50.346522 (+ 3015us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.346530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.347864 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 06:58:50.348005 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.348011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.348173 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.348335 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:50.349260 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.350383 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.352880 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.354009 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.354127 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.354303 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.354576 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.354736 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2874,"thread_start_us":2786,"threads_started":1}
I20260620 06:58:50.355708 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.343003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40209 (local address 127.0.0.1:59264)
0620 06:58:50.347711 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.347729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.347844 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.348541 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.348546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.348572 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:50.349075 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.349087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.350572 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.350577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.352700 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.352712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.354186 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.354192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.354347 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.354379 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.354459 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.355387 (+ 928us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4561,"mutex_wait_us":826,"thread_start_us":4402,"threads_started":1}
I20260620 06:58:50.357257 11115 tablet_service.cc:1467] Processing CreateTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (DEFAULT_TABLE table=test-table [id=e663b614f65a45f9887db84f49f6caf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:50.358458 11115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:50.358865 10848 tablet_service.cc:1467] Processing CreateTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (DEFAULT_TABLE table=test-table [id=e663b614f65a45f9887db84f49f6caf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:50.358865 10982 tablet_service.cc:1467] Processing CreateTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (DEFAULT_TABLE table=test-table [id=e663b614f65a45f9887db84f49f6caf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:58:50.360142 10982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:50.360146 10848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:58:50.369098 11198 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Bootstrap starting.
I20260620 06:58:50.371984 11198 tablet_bootstrap.cc:654] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:50.372915 11198 log.cc:824] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:50.373478 11199 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Bootstrap starting.
I20260620 06:58:50.374238 11200 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Bootstrap starting.
I20260620 06:58:50.377089 11199 tablet_bootstrap.cc:654] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:50.377615 11200 tablet_bootstrap.cc:654] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Neither blocks nor log segments found. Creating new log.
I20260620 06:58:50.378209 11199 log.cc:824] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:50.378641 11200 log.cc:824] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Log is configured to *not* fsync() on all Append() calls
I20260620 06:58:50.379694 11198 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: No bootstrap required, opened a new log
I20260620 06:58:50.379938 11198 ts_tablet_manager.cc:1397] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260620 06:58:50.381424 11200 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: No bootstrap required, opened a new log
I20260620 06:58:50.381481 11199 tablet_bootstrap.cc:492] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: No bootstrap required, opened a new log
I20260620 06:58:50.381685 11200 ts_tablet_manager.cc:1397] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 06:58:50.381776 11199 ts_tablet_manager.cc:1397] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 06:58:50.391750 11198 raft_consensus.cc:348] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.392266 11198 raft_consensus.cc:374] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:50.392393 11198 raft_consensus.cc:729] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76db6c2f674f400d8662da9301a95437, State: Initialized, Role: FOLLOWER
I20260620 06:58:50.393257 11198 consensus_queue.cc:260] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.393114 11200 raft_consensus.cc:348] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.393554 11200 raft_consensus.cc:374] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:50.393678 11200 raft_consensus.cc:729] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ec81ccc4e454c83037e1c8dfb7744, State: Initialized, Role: FOLLOWER
I20260620 06:58:50.393890 11199 raft_consensus.cc:348] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.394351 11199 raft_consensus.cc:374] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:58:50.394481 11199 raft_consensus.cc:729] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a148d1be2f4e149286508ecd3f507e, State: Initialized, Role: FOLLOWER
I20260620 06:58:50.394408 11200 consensus_queue.cc:260] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.395186 11199 consensus_queue.cc:260] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.395720 11180 heartbeater.cc:503] Master 127.0.97.62:32869 was elected leader, sending a full tablet report...
I20260620 06:58:50.397468 11198 ts_tablet_manager.cc:1428] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260620 06:58:50.402565 11200 ts_tablet_manager.cc:1428] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
I20260620 06:58:50.402951 11047 heartbeater.cc:503] Master 127.0.97.62:32869 was elected leader, sending a full tablet report...
I20260620 06:58:50.404249 11199 ts_tablet_manager.cc:1428] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Time spent starting tablet: real 0.022s user 0.015s sys 0.002s
I20260620 06:58:50.433652 11205 raft_consensus.cc:482] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:58:50.433943 11205 raft_consensus.cc:504] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.435557 11205 leader_election.cc:288] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76db6c2f674f400d8662da9301a95437 (127.0.97.3:44803), 25a148d1be2f4e149286508ecd3f507e (127.0.97.1:39931)
I20260620 06:58:50.442142 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.435984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54425 (local address 127.0.97.3:44803)
0620 06:58:50.436196 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.436213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.436885 (+ 672us) server_negotiation.cc:407] Connection header received
0620 06:58:50.437063 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.437067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.437127 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.437208 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:50.437845 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.438568 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.440624 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.441360 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.441469 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.441567 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.441871 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.441965 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:58:50.442337 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.435968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44803 (local address 127.0.97.2:54425)
0620 06:58:50.436772 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.436794 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.436940 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.437242 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.437246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.437269 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:50.437719 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.437729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.438681 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.438686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.440481 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.440494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.441550 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.441556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.441701 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.441735 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.441823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.441945 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":81,"thread_start_us":390,"threads_started":1}
I20260620 06:58:50.443492 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.435972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39931 (local address 127.0.97.2:45777)
0620 06:58:50.436839 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 06:58:50.436852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:58:50.436976 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:58:50.437256 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:58:50.437260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:58:50.437281 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:58:50.437677 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.437690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.438723 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.438730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.440479 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:58:50.440493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.442937 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.442943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:58:50.443086 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.443109 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:58:50.443178 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:58:50.443258 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":219,"thread_start_us":397,"threads_started":1}
I20260620 06:58:50.443580 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.436090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45777 (local address 127.0.97.1:39931)
0620 06:58:50.436311 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.436318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.436936 (+ 618us) server_negotiation.cc:407] Connection header received
0620 06:58:50.437098 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.437101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.437158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.437228 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:58:50.437822 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.438602 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.441642 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.442758 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.442861 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.442927 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.443241 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.443366 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:58:50.443456 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874d3f4ae1ec43e6a9fb9f070fdd24ba" candidate_uuid: "356ec81ccc4e454c83037e1c8dfb7744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76db6c2f674f400d8662da9301a95437" is_pre_election: true
I20260620 06:58:50.444015 11135 raft_consensus.cc:2454] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ec81ccc4e454c83037e1c8dfb7744 in term 0.
I20260620 06:58:50.444613 10935 leader_election.cc:302] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ec81ccc4e454c83037e1c8dfb7744, 76db6c2f674f400d8662da9301a95437; no voters:
I20260620 06:58:50.444589 10868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874d3f4ae1ec43e6a9fb9f070fdd24ba" candidate_uuid: "356ec81ccc4e454c83037e1c8dfb7744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a148d1be2f4e149286508ecd3f507e" is_pre_election: true
I20260620 06:58:50.445027 11205 raft_consensus.cc:2781] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:58:50.445080 10868 raft_consensus.cc:2454] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356ec81ccc4e454c83037e1c8dfb7744 in term 0.
I20260620 06:58:50.445169 11205 raft_consensus.cc:482] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:58:50.445266 11205 raft_consensus.cc:3037] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:50.447623 11205 raft_consensus.cc:504] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.448195 11205 leader_election.cc:288] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [CANDIDATE]: Term 1 election: Requested vote from peers 76db6c2f674f400d8662da9301a95437 (127.0.97.3:44803), 25a148d1be2f4e149286508ecd3f507e (127.0.97.1:39931)
I20260620 06:58:50.448565 11135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874d3f4ae1ec43e6a9fb9f070fdd24ba" candidate_uuid: "356ec81ccc4e454c83037e1c8dfb7744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76db6c2f674f400d8662da9301a95437"
I20260620 06:58:50.448689 10868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "874d3f4ae1ec43e6a9fb9f070fdd24ba" candidate_uuid: "356ec81ccc4e454c83037e1c8dfb7744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25a148d1be2f4e149286508ecd3f507e"
I20260620 06:58:50.448729 11135 raft_consensus.cc:3037] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:50.448873 10868 raft_consensus.cc:3037] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:58:50.450909 11135 raft_consensus.cc:2454] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ec81ccc4e454c83037e1c8dfb7744 in term 1.
I20260620 06:58:50.451074 10868 raft_consensus.cc:2454] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356ec81ccc4e454c83037e1c8dfb7744 in term 1.
I20260620 06:58:50.451260 10935 leader_election.cc:302] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356ec81ccc4e454c83037e1c8dfb7744, 76db6c2f674f400d8662da9301a95437; no voters:
I20260620 06:58:50.451539 11205 raft_consensus.cc:2781] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:58:50.451918 11205 raft_consensus.cc:686] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 1 LEADER]: Becoming Leader. State: Replica: 356ec81ccc4e454c83037e1c8dfb7744, State: Running, Role: LEADER
I20260620 06:58:50.452373 11205 consensus_queue.cc:237] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } }
I20260620 06:58:50.456717 10723 catalog_manager.cc:5495] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 356ec81ccc4e454c83037e1c8dfb7744 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "356ec81ccc4e454c83037e1c8dfb7744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "356ec81ccc4e454c83037e1c8dfb7744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40209 } health_report { overall_health: HEALTHY } } }
I20260620 06:58:50.481061 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.476206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.0.97.2:40209)
0620 06:58:50.476410 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.476414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.476430 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:58:50.476541 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.476544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.476615 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.476724 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:58:50.477031 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.477715 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.478949 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.479425 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.479528 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.479962 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:58:50.480550 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:58:50.480680 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.480719 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.480781 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 06:58:50.488876 10868 raft_consensus.cc:1261] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 1 FOLLOWER]: Refusing update from remote peer 356ec81ccc4e454c83037e1c8dfb7744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:50.488940 11135 raft_consensus.cc:1261] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 1 FOLLOWER]: Refusing update from remote peer 356ec81ccc4e454c83037e1c8dfb7744: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:58:50.489789 11205 consensus_queue.cc:1035] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25a148d1be2f4e149286508ecd3f507e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:50.490068 11209 consensus_queue.cc:1035] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76db6c2f674f400d8662da9301a95437" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:58:50.501667 11214 mvcc.cc:205] Tried to move back new op lower bound from 7298821040073129984 to 7298821039936544768. Current Snapshot: MvccSnapshot[applied={T|T < 7298821040073129984}]
I20260620 06:58:50.504370 11212 mvcc.cc:205] Tried to move back new op lower bound from 7298821040073129984 to 7298821039936544768. Current Snapshot: MvccSnapshot[applied={T|T < 7298821040073129984}]
I20260620 06:58:50.517519 10723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:50.517768 10723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:58:50.521167 10723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c86e9e2b405c489a9edb95b47d488035: Sending DeleteTablet for 3 replicas of tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba
I20260620 06:58:50.522311 10982 tablet_service.cc:1514] Processing DeleteTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59264
I20260620 06:58:50.522503 11115 tablet_service.cc:1514] Processing DeleteTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260620 06:58:50.523027 10848 tablet_service.cc:1514] Processing DeleteTablet for tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:58:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39056
I20260620 06:58:50.523813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10785
I20260620 06:58:50.532094 11224 tablet_replica.cc:330] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: stopping tablet replica
I20260620 06:58:50.532608 11224 raft_consensus.cc:2229] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:50.533407 11224 raft_consensus.cc:2258] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:50.535820 11224 ts_tablet_manager.cc:1905] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:50.536409 11225 tablet_replica.cc:330] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: stopping tablet replica
I20260620 06:58:50.536875 11225 raft_consensus.cc:2229] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:58:50.537480 11225 raft_consensus.cc:2258] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:50.538975 11223 tablet_replica.cc:330] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: stopping tablet replica
I20260620 06:58:50.540832 11225 ts_tablet_manager.cc:1905] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 06:58:50 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:58:50.542375 11223 raft_consensus.cc:2229] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:58:50.543350 11223 raft_consensus.cc:2258] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:58:50.544979 11224 ts_tablet_manager.cc:1918] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:50.545192 11224 log.cc:1196] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/874d3f4ae1ec43e6a9fb9f070fdd24ba
I20260620 06:58:50.545575 11223 ts_tablet_manager.cc:1905] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:58:50.545743 11224 ts_tablet_manager.cc:1939] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 76db6c2f674f400d8662da9301a95437: Deleting consensus metadata
I20260620 06:58:50.547554 10708 catalog_manager.cc:4841] TS 76db6c2f674f400d8662da9301a95437 (127.0.97.3:44803): tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (table test-table [id=e663b614f65a45f9887db84f49f6caf5]) successfully deleted
I20260620 06:58:50.548624 11225 ts_tablet_manager.cc:1918] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:50.548774 11225 log.cc:1196] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/874d3f4ae1ec43e6a9fb9f070fdd24ba
I20260620 06:58:50.549283 11225 ts_tablet_manager.cc:1939] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 25a148d1be2f4e149286508ecd3f507e: Deleting consensus metadata
I20260620 06:58:50.551756 10709 catalog_manager.cc:4841] TS 25a148d1be2f4e149286508ecd3f507e (127.0.97.1:39931): tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (table test-table [id=e663b614f65a45f9887db84f49f6caf5]) successfully deleted
I20260620 06:58:50.552636 11223 ts_tablet_manager.cc:1918] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:58:50.552816 11223 log.cc:1196] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/874d3f4ae1ec43e6a9fb9f070fdd24ba
I20260620 06:58:50.553372 11223 ts_tablet_manager.cc:1939] T 874d3f4ae1ec43e6a9fb9f070fdd24ba P 356ec81ccc4e454c83037e1c8dfb7744: Deleting consensus metadata
I20260620 06:58:50.555039 10708 catalog_manager.cc:4841] TS 356ec81ccc4e454c83037e1c8dfb7744 (127.0.97.2:40209): tablet 874d3f4ae1ec43e6a9fb9f070fdd24ba (table test-table [id=e663b614f65a45f9887db84f49f6caf5]) successfully deleted
I20260620 06:58:50.557447 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.524763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.0.97.1:39931)
0620 06:58:50.524922 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.524928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.531809 (+ 6881us) server_negotiation.cc:407] Connection header received
0620 06:58:50.532168 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.532174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.532298 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.532373 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:50.534170 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.535294 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.537027 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.537658 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.537766 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.550762 (+ 12996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:50.550775 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:50.550795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:50.550844 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:50.554713 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.555516 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.555525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.555529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.555657 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.556009 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.556014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.556018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.556382 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:50.556645 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.556911 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.557102 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:58:50.558068 10908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:50.645279 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10785
I20260620 06:58:50.659799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10919
Jun 20 06:58:50 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:58:50.677054 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.660554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.0.97.2:40209)
0620 06:58:50.660725 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.660730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.660749 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:58:50.660860 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.660866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.660941 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.661044 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:50.662125 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.663445 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.664784 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.665461 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.665587 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.669962 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:50.669981 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:50.670002 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:50.670048 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:50.673925 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.674845 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.674855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.674859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.675010 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.675415 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.675420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.675424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.675854 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:50.676186 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.676439 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.676663 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:58:50.677771 11042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:50.802947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10919
I20260620 06:58:50.819469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 11052
Jun 20 06:58:50 dist-test-slave-mz20 krb5kdc[10679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:58:50.833614 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:58:50.820194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.0.97.3:44803)
0620 06:58:50.820376 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:58:50.820383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:58:50.820484 (+ 101us) server_negotiation.cc:407] Connection header received
0620 06:58:50.820678 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:58:50.820682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:58:50.820773 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:58:50.820859 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:58:50.822108 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.822875 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.824115 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:58:50.824494 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:58:50.824603 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:58:50.828581 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:58:50.828596 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:58:50.828611 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:58:50.828649 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:58:50.831077 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.831700 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.831728 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.831730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.831860 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:58:50.832196 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:58:50.832199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:58:50.832201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:58:50.832539 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:58:50.832811 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 06:58:50.833093 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 06:58:50.833301 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:58:50.834204 11175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:50.921551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 11052
I20260620 06:58:50.934187 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 10692
I20260620 06:58:50.935254 10754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:58:51.040819 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 10692
2026-06-20T06:58:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3933 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:58:51 dist-test-slave-mz20 krb5kdc[11233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:58:51 dist-test-slave-mz20 krb5kdc[11233](info): set up 2 sockets
Jun 20 06:58:51 dist-test-slave-mz20 krb5kdc[11233](info): commencing operation
krb5kdc: starting...
W20260620 06:58:52.047160 388 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:58:52 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:58:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 06:58:52.337530 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8UMGcc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/postgres -l logfile start
2026-06-20 06:58:54.687 UTC [11263] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:58:54.687 UTC [11263] LOG: listening on IPv4 address "127.0.97.20", port 54199
2026-06-20 06:58:54.690 UTC [11263] LOG: listening on Unix socket "/tmp/.s.PGSQL.54199"
2026-06-20 06:58:54.715 UTC [11266] LOG: database system was shut down at 2026-06-20 06:58:53 UTC
2026-06-20 06:58:54.718 UTC [11263] LOG: database system is ready to accept connections
I20260620 06:58:55.618906 388 mini_postgres.cc:96] Postgres bound to 54199
2026-06-20 06:58:55.635 UTC [11274] FATAL: database "slave" does not exist
127.0.97.20:54199 - accepting connections
I20260620 06:58:55.636229 388 mini_ranger.cc:161] Starting Ranger...
I20260620 06:58:55.661661 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:58:56.119294 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:58:56.119462 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin
2026-06-20 06:58:56,392 [I] DB FLAVOR :POSTGRES
2026-06-20 06:58:56,392 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:58:56,392 [I] Checking connection..
2026-06-20 06:58:56,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:58:56,653 [I] Checking connection passed.
2026-06-20 06:58:56,653 [I] --------- Verifying version history table ---------
2026-06-20 06:58:56,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:58:56,898 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:58:56,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:58:57,136 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:58:57,136 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:58:57,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:58:57.383 UTC [11338] WARNING: there is no transaction in progress
2026-06-20 06:58:57,389 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:58:57,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:58:57,645 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:58:57,645 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:58:57,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:58:57,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:58:58,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mz20.c.gcp-upstream.internal','N') ;"
2026-06-20 06:58:58,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:58:58,601 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:58:58,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:58:58,854 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:58:58,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:58:59,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:58:59,344 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:58:59,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:59:00.311 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.327 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.346 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.362 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.378 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.395 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.454 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.462 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.479 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.489 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.499 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.511 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.520 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.527 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.533 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:00.540 UTC [11446] WARNING: there is no transaction in progress
2026-06-20 06:59:01,154 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:59:01,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:59:01,403 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:59:01,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:59:01,661 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:59:01,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:59:01,916 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:59:01,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:59:02,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:59:02,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mz20.c.gcp-upstream.internal';"
2026-06-20 06:59:02,650 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:59:02,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:59:02,879 [I] DB_PATCHES have already been applied
I20260620 06:59:02.884778 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8UMGcc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:59:02.884928 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 06:59:02.898870 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:36731
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:59:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:59:03.916069 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.280s user 0.003s sys 0.035s
Jun 20, 2026 6:59:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:59:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:59:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:59:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:59:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:59:24.450520 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:59:24 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938764, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:59:24.619217 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:59:24.641968 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:59:25.481782 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:59:25.481911 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms
I20260620 06:59:25.482048 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54199
2026-06-20 06:59:25,866 [I] DB FLAVOR :POSTGRES
2026-06-20 06:59:25,867 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:59:25,867 [I] Checking connection
2026-06-20 06:59:25,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:59:26,099 [I] connection success
2026-06-20 06:59:26,099 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:59:26,099 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:59:26,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:59:26,343 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:59:26,344 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:59:26,344 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:59:26,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:59:26,593 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:59:27.039609 388 mini_ranger_kms.cc:310] Created kms service
I20260620 06:59:27.204857 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:59:27.247134 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:59:27 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938764, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 06:59:27.642784 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:59:27.643049 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:59:27.643098 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 06:59:27.658529 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11626
I20260620 06:59:27.658742 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:46839
Jun 20, 2026 6:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:59:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:59:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 06:59:29 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781938769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:59:29.750842 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.132s user 0.091s sys 0.313s
I20260620 06:59:29.751086 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:59:29.751267 388 mini_ranger_kms.cc:401] 127.0.97.20:46839/kms/v1/keys
Jun 20 06:59:30 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 06:59:30 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938769, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 06:59:33 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 06:59:33.124512 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45645
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45645
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46839/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:59:33.453486 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:59:33.453780 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:59:33.453908 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:59:33.460908 11677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:59:33.461016 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:59:33.461091 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:59:33.461161 11677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:59:33.461221 11677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:59:33.471385 11677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:46839/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45645
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:59:33.473062 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:59:33.474673 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:59:33.486487 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:33.486929 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:33.487428 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:59:33.488441 11677 server_base.cc:1034] running on GCE node
I20260620 06:59:33.489157 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:59:33.490526 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:59:33.491770 11677 hybrid_clock.cc:648] HybridClock initialized: now 1781938773491723 us; error 31 us; skew 500 ppm
I20260620 06:59:33.492084 11677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:59:33 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:59:33.494982 11677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 06:59:33.496294 11677 webserver.cc:460] Webserver started at http://127.0.97.62:44409/ using document root <none> and password file <none>
I20260620 06:59:33.496930 11677 fs_manager.cc:359] Metadata directory not provided
I20260620 06:59:33.497045 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:59:33.497309 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:59:33 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 06:59:33.563602 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c"
format_stamp: "Formatted at 2026-06-20 06:59:33 on dist-test-slave-mz20"
server_key: "77a6c8253f15d235c9364557f5a0f130"
server_key_iv: "a4338ca1068878190c76805672d92b48"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:33.564380 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c"
format_stamp: "Formatted at 2026-06-20 06:59:33 on dist-test-slave-mz20"
server_key: "77a6c8253f15d235c9364557f5a0f130"
server_key_iv: "a4338ca1068878190c76805672d92b48"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:33.569803 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 06:59:33.589658 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:33.591459 11677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:59:33.591682 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c"
format_stamp: "Formatted at 2026-06-20 06:59:33 on dist-test-slave-mz20"
server_key: "77a6c8253f15d235c9364557f5a0f130"
server_key_iv: "a4338ca1068878190c76805672d92b48"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:33.591941 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:59:33.613730 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:59:33.618590 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:59:33.618917 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:59:33.638516 11677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45645
I20260620 06:59:33.638509 11743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45645 every 8 connection(s)
I20260620 06:59:33.640203 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 06:59:33.644030 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:59:33.650005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 11677
I20260620 06:59:33.650214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 06:59:33.650987 388 external_mini_cluster.cc:1428] Setting key 5d8ce20f153ff81fe31c6f7ddf8adb1a
I20260620 06:59:33.655800 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: Bootstrap starting.
I20260620 06:59:33.660655 11744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: Neither blocks nor log segments found. Creating new log.
I20260620 06:59:33.662179 11744 log.cc:824] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:59:33.670888 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: No bootstrap required, opened a new log
I20260620 06:59:33.680279 11744 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } }
I20260620 06:59:33.680680 11744 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:59:33.680819 11744 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf86bbbc87574ff5a90ed3cfebdffb6c, State: Initialized, Role: FOLLOWER
I20260620 06:59:33.681547 11744 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } }
I20260620 06:59:33.681823 11744 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:59:33.681957 11744 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:59:33.682107 11744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 0 FOLLOWER]: Advancing to term 1
Jun 20 06:59:33 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:59:33.684816 11744 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } }
I20260620 06:59:33.685508 11744 leader_election.cc:302] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf86bbbc87574ff5a90ed3cfebdffb6c; no voters:
I20260620 06:59:33.686228 11744 leader_election.cc:288] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:59:33.686375 11749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:59:33.689595 11744 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:59:33.692382 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:33.653161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.97.62:45645)
0620 06:59:33.667915 (+ 14754us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:33.667932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:33.667977 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:59:33.670295 (+ 2318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:33.670327 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:33.670899 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:33.671385 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:33.672872 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:33.674992 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:33.676890 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:33.678004 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:33.678199 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:33.683534 (+ 5335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:33.683546 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:33.683569 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:33.683630 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:33.687651 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:33.688483 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:33.688492 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:33.688500 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:33.688658 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:33.689041 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:33.689047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:33.689051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:33.689745 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:33.690046 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:33.690472 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 06:59:33.690930 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14446,"thread_start_us":15328,"threads_started":1}
I20260620 06:59:33.695444 11749 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [term 1 LEADER]: Becoming Leader. State: Replica: cf86bbbc87574ff5a90ed3cfebdffb6c, State: Running, Role: LEADER
I20260620 06:59:33.696105 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } }
I20260620 06:59:33.700601 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf86bbbc87574ff5a90ed3cfebdffb6c. Latest consensus state: current_term: 1 leader_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } } }
I20260620 06:59:33.700572 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf86bbbc87574ff5a90ed3cfebdffb6c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45645 } } }
I20260620 06:59:33.701135 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [sys.catalog]: This master's current role is: LEADER
I20260620 06:59:33.701141 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c [sys.catalog]: This master's current role is: LEADER
I20260620 06:59:33.708909 11754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:59:33.717473 11754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:59:33.718187 11744 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:59:33.743618 11754 catalog_manager.cc:1347] Generated new cluster ID: 56b8892a9ce3468fa25c0f90f3ecfb6f
I20260620 06:59:33.743923 11754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:59:33.771715 11754 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:59:33.773711 11754 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:59:33.785210 11754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: Generated new TSK 0
I20260620 06:59:33.786240 11754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 06:59:34.544622 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:59:34 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 06:59:34.942075 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:59:34.942351 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:59:34.942471 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:59:34.950675 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:59:34.950800 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:59:34.950954 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 06:59:34.961469 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:46839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:59:34.962834 11789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:59:34.964177 11789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:59:34.979422 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:34.980751 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:34.980940 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:59:34.981628 11789 server_base.cc:1034] running on GCE node
I20260620 06:59:34.982242 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:59:34.983242 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:59:34.984436 11789 hybrid_clock.cc:648] HybridClock initialized: now 1781938774984368 us; error 60 us; skew 500 ppm
I20260620 06:59:34.984730 11789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:59:34 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:59:34.989161 11789 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 06:59:34.998656 11789 webserver.cc:460] Webserver started at http://127.0.97.1:36565/ using document root <none> and password file <none>
I20260620 06:59:34.999616 11789 fs_manager.cc:359] Metadata directory not provided
I20260620 06:59:34.999835 11789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:59:35.000254 11789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 06:59:35.029098 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "7703b2381c6042d283f0f24a33c61614"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "ebb4b5e5a5ee352ea3d8d60a14844667"
server_key_iv: "ebee77d40a4c4c72613c64052784538f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.030365 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7703b2381c6042d283f0f24a33c61614"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "ebb4b5e5a5ee352ea3d8d60a14844667"
server_key_iv: "ebee77d40a4c4c72613c64052784538f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.050770 11789 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.013s sys 0.004s
I20260620 06:59:35.073974 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:35.075696 11789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:59:35.075892 11789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "7703b2381c6042d283f0f24a33c61614"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "ebb4b5e5a5ee352ea3d8d60a14844667"
server_key_iv: "ebee77d40a4c4c72613c64052784538f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.076126 11789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:59:35.122628 11789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:59:35.127493 11789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:59:35.127775 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:59:35.128778 11789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:59:35.130564 11789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:59:35.130650 11789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:35.130769 11789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:59:35.130820 11789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:35.204205 11789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45211
I20260620 06:59:35.204206 11921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45211 every 8 connection(s)
I20260620 06:59:35.205662 11789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 06:59:35.212522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 11789
I20260620 06:59:35.212713 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 06:59:35.213493 388 external_mini_cluster.cc:1428] Setting key c19e9fcf8fc41f0489f2fc203eae6c4d
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:59:35.236802 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:35.211482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45645 (local address 127.0.97.1:55891)
0620 06:59:35.212374 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:35.212428 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:35.216585 (+ 4157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:35.217807 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:35.217820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:35.218429 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:59:35.219261 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:35.219286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:35.220703 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:35.220708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:35.222599 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:35.222615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:35.223888 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:35.223895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:35.224183 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:35.224990 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:59:35.225017 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:59:35.227805 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:59:35.232152 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:35.232161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:35.232188 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:35.232760 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:35.233210 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:35.233215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:35.233218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:35.233450 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:35.234107 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:59:35.234115 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:59:35.234582 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 06:59:35.234862 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 06:59:35.235410 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":4710,"threads_started":1}
I20260620 06:59:35.238047 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:35.214192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55891 (local address 127.0.97.62:45645)
0620 06:59:35.214998 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:35.215005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:35.215028 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:59:35.217001 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:35.217008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:35.217097 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:35.217178 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:35.219459 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:35.220487 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:35.222770 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:35.223322 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:35.223425 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:35.228078 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:35.228086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:35.228091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:35.228142 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:35.231999 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:35.232913 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:35.232920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:35.232923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:35.233067 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:35.233574 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:35.233579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:35.233582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:35.233911 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:35.234038 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:35.237523 (+ 3485us) server_negotiation.cc:299] Negotiation successful
0620 06:59:35.237737 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":2951,"threads_started":1}
I20260620 06:59:35.239712 11922 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45645
I20260620 06:59:35.240061 11922 heartbeater.cc:464] Registering TS with master...
I20260620 06:59:35.240926 11922 heartbeater.cc:511] Master 127.0.97.62:45645 requested a full tablet report, sending...
I20260620 06:59:35.243381 11709 ts_manager.cc:194] Registered new tserver with Master: 7703b2381c6042d283f0f24a33c61614 (127.0.97.1:45211)
I20260620 06:59:35.245479 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55891
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 06:59:35.275722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 06:59:35.769518 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:59:35.770025 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:59:35.770246 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:59:35.783023 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:59:35.783248 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:59:35.783483 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 06:59:35.797960 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:46839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:59:35.799918 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:59:35.801595 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:59:35.834067 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:35.840946 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:35.841734 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:59:35.842810 11928 server_base.cc:1034] running on GCE node
I20260620 06:59:35.843637 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:59:35.854889 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:59:35.857131 11928 hybrid_clock.cc:648] HybridClock initialized: now 1781938775857008 us; error 159 us; skew 500 ppm
I20260620 06:59:35.857599 11928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:59:35.867092 11928 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 06:59:35.879444 11928 webserver.cc:460] Webserver started at http://127.0.97.2:43865/ using document root <none> and password file <none>
I20260620 06:59:35.880462 11928 fs_manager.cc:359] Metadata directory not provided
I20260620 06:59:35.880612 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:59:35.880980 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:59:35 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 06:59:35.902263 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "19be3001db774c988f9bc72f39b402ee"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "d25dd074fbab3601d2a7836f4bafe068"
server_key_iv: "f3f74f366ce04e6bd10ef0fdedfdf00f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.903683 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "19be3001db774c988f9bc72f39b402ee"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "d25dd074fbab3601d2a7836f4bafe068"
server_key_iv: "f3f74f366ce04e6bd10ef0fdedfdf00f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.917071 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260620 06:59:35.938340 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:35.940351 11928 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 06:59:35.941045 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "19be3001db774c988f9bc72f39b402ee"
format_stamp: "Formatted at 2026-06-20 06:59:35 on dist-test-slave-mz20"
server_key: "d25dd074fbab3601d2a7836f4bafe068"
server_key_iv: "f3f74f366ce04e6bd10ef0fdedfdf00f"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:35.941344 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:59:35.958410 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:59:35.962411 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:59:35.962735 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:59:35.963693 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:59:35.965157 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:59:35.965227 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:35.965337 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:59:35.965384 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:36.010619 11928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40649
I20260620 06:59:36.010671 12064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40649 every 8 connection(s)
I20260620 06:59:36.011910 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 06:59:36.018908 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 11928
I20260620 06:59:36.019093 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 06:59:36.019641 388 external_mini_cluster.cc:1428] Setting key f877fa5ed1811c2bf88da9456185ca42
Jun 20 06:59:36 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:59:36.034605 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.014502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56123 (local address 127.0.97.62:45645)
0620 06:59:36.014951 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.014955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.016097 (+ 1142us) server_negotiation.cc:407] Connection header received
0620 06:59:36.018098 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.018104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.018204 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.018309 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:36.019834 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.020673 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.023092 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.023619 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.023741 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.028100 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:36.028108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:36.028113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:36.028163 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:36.031098 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.031833 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.031838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.031840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.032005 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.032806 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.032809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.032812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.033086 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:36.033215 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.033968 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.034207 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260620 06:59:36.035676 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.015210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45645 (local address 127.0.97.2:56123)
0620 06:59:36.015953 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.015990 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.017834 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.018587 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.018599 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.019053 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:59:36.019659 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.019676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.020844 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.020850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.022931 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.022950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.023893 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.023903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.024259 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.025143 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:59:36.025188 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:59:36.027815 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:59:36.031246 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:36.031261 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:36.031276 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:36.031694 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:36.032473 (+ 779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:36.032484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:36.032486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:36.032679 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:36.033212 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:59:36.033220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:59:36.033590 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.033842 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.034389 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":323,"threads_started":1}
I20260620 06:59:36.037086 12065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45645
I20260620 06:59:36.037375 12065 heartbeater.cc:464] Registering TS with master...
I20260620 06:59:36.038070 12065 heartbeater.cc:511] Master 127.0.97.62:45645 requested a full tablet report, sending...
I20260620 06:59:36.039454 11709 ts_manager.cc:194] Registered new tserver with Master: 19be3001db774c988f9bc72f39b402ee (127.0.97.2:40649)
I20260620 06:59:36.040311 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56123
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 06:59:36.077077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46839/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:59:36.249481 11922 heartbeater.cc:503] Master 127.0.97.62:45645 was elected leader, sending a full tablet report...
W20260620 06:59:36.462025 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:59:36.462297 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:59:36.462414 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:59:36.470371 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:59:36.470485 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:59:36.470670 12071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 06:59:36.480626 12071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:46839/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 06:59:36.481961 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:59:36.483310 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:59:36.496326 12079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:36.496307 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:59:36.496518 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:59:36.497069 12071 server_base.cc:1034] running on GCE node
I20260620 06:59:36.497633 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:59:36.498586 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:59:36.499811 12071 hybrid_clock.cc:648] HybridClock initialized: now 1781938776499741 us; error 51 us; skew 500 ppm
I20260620 06:59:36.500123 12071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 06:59:36 dist-test-slave-mz20 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:59:36.503178 12071 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 06:59:36.504465 12071 webserver.cc:460] Webserver started at http://127.0.97.3:41323/ using document root <none> and password file <none>
I20260620 06:59:36.505075 12071 fs_manager.cc:359] Metadata directory not provided
I20260620 06:59:36.505182 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:59:36.505430 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:59:36 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 06:59:36.521150 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "4006044e439542caa6423e7bd79e89a2"
format_stamp: "Formatted at 2026-06-20 06:59:36 on dist-test-slave-mz20"
server_key: "7dbc4973fd63869d95fa5710569050d3"
server_key_iv: "33ccf31c74d158870e6574b89753a4df"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:36.522032 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4006044e439542caa6423e7bd79e89a2"
format_stamp: "Formatted at 2026-06-20 06:59:36 on dist-test-slave-mz20"
server_key: "7dbc4973fd63869d95fa5710569050d3"
server_key_iv: "33ccf31c74d158870e6574b89753a4df"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:36.527572 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 06:59:36.544816 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:36.546725 12071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:59:36.546931 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "4006044e439542caa6423e7bd79e89a2"
format_stamp: "Formatted at 2026-06-20 06:59:36 on dist-test-slave-mz20"
server_key: "7dbc4973fd63869d95fa5710569050d3"
server_key_iv: "33ccf31c74d158870e6574b89753a4df"
server_key_version: "kuduclusterkey@0"
I20260620 06:59:36.547180 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:59:36.588341 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:59:36.592455 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:59:36.592710 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:59:36.593600 12071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:59:36.594995 12071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:59:36.595067 12071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:36.595150 12071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:59:36.595193 12071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:59:36.627046 12071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36847
I20260620 06:59:36.627115 12198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36847 every 8 connection(s)
I20260620 06:59:36.628290 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 06:59:36.634734 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 12071
I20260620 06:59:36.634895 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 06:59:36.635296 388 external_mini_cluster.cc:1428] Setting key 57966359d749acb7bfd07d3a7cba7af9
Jun 20 06:59:36 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 06:59:36.649503 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.631182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41919 (local address 127.0.97.62:45645)
0620 06:59:36.631601 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.631605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.632529 (+ 924us) server_negotiation.cc:407] Connection header received
0620 06:59:36.634410 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.634416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.634500 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.634641 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:36.636091 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.636963 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.639186 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.639630 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.639727 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.643411 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:36.643417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:36.643420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:36.643468 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:36.646423 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.647187 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.647193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.647196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.647362 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.647830 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.647833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.647835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.648096 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:36.648238 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.648924 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.649131 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":207,"threads_started":1}
I20260620 06:59:36.650848 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.631644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45645 (local address 127.0.97.3:41919)
0620 06:59:36.632356 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.632405 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.634146 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.634887 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.634900 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.635336 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:59:36.635927 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.635944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.637136 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.637142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.639056 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.639069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.639820 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.639826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.640098 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.640764 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:59:36.640795 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:59:36.643186 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:59:36.646602 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:36.646613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:36.646631 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:36.647070 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:36.647468 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:59:36.647473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:59:36.647476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:59:36.647688 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:59:36.648211 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:59:36.648219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:59:36.648572 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.648818 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.649329 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":296,"threads_started":1}
I20260620 06:59:36.652069 12199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45645
I20260620 06:59:36.652343 12199 heartbeater.cc:464] Registering TS with master...
I20260620 06:59:36.653033 12199 heartbeater.cc:511] Master 127.0.97.62:45645 requested a full tablet report, sending...
I20260620 06:59:36.654373 11709 ts_manager.cc:194] Registered new tserver with Master: 4006044e439542caa6423e7bd79e89a2 (127.0.97.3:36847)
I20260620 06:59:36.655202 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41919
I20260620 06:59:36.663196 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:59:36.679769 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.667749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.0.97.62:45645)
0620 06:59:36.667911 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.667917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.668321 (+ 404us) server_negotiation.cc:407] Connection header received
0620 06:59:36.668457 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.668461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.668539 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.668643 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:36.669927 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.670707 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.672051 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.672484 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.672586 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.674974 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:36.674980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:36.674983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:36.675016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:36.677455 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.678075 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.678079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.678081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.678191 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:36.678580 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:36.678585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:36.678587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:36.678855 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:36.679022 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.679307 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.679466 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:59:36.684016 11709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:59:36.690903 11709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:59:36.719178 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.703478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.0.97.3:36847)
0620 06:59:36.710253 (+ 6775us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.710260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.710289 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:59:36.710380 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.710388 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.710679 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.711612 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:59:36.711874 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.713649 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.716464 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.717911 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.718039 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.718284 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.718641 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.718851 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6622,"thread_start_us":6484,"threads_started":1}
I20260620 06:59:36.719238 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.703964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40649 (local address 127.0.0.1:42038)
0620 06:59:36.704944 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.704962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.705063 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.711615 (+ 6552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.711621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.711652 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:59:36.712067 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.712078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.713852 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.713857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.716591 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.716604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.718288 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.718295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.718432 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.718462 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.718522 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.718817 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":51,"thread_start_us":8389,"threads_started":1}
I20260620 06:59:36.719322 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.703267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36847 (local address 127.0.0.1:48608)
0620 06:59:36.704021 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.704071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.704247 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.710948 (+ 6701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.710956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.710988 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:59:36.711400 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.711414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.713790 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.713795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.716300 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.716311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.718099 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.718105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.718297 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.718351 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.718590 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.718816 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":131,"thread_start_us":276,"threads_started":1}
I20260620 06:59:36.720372 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.704766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.0.97.2:40649)
0620 06:59:36.710962 (+ 6196us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.710970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.710996 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:59:36.711084 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.711091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.711360 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.711615 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:59:36.712281 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.713741 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.716776 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.718085 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.719771 (+ 1686us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.719959 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.720011 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.720122 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6039,"thread_start_us":5936,"threads_started":1}
I20260620 06:59:36.720688 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.704393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45211 (local address 127.0.0.1:44402)
0620 06:59:36.712595 (+ 8202us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.712612 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.712817 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.713623 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.713626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.713644 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:59:36.714063 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.714074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.716228 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.716234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.717991 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.718004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.720013 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.720019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.720170 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.720212 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.720299 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.720410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8082,"thread_start_us":1362,"threads_started":1}
I20260620 06:59:36.721596 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.705309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.0.97.1:45211)
0620 06:59:36.705916 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.705926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.712718 (+ 6792us) server_negotiation.cc:407] Connection header received
0620 06:59:36.713017 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.713026 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.713280 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.713417 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:59:36.714247 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.716062 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.718161 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.719600 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.719673 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.719835 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.720363 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.720521 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":255,"threads_started":1}
I20260620 06:59:36.723737 12000 tablet_service.cc:1467] Processing CreateTablet for tablet c5cc826d0e0846d3a94156608c0570f5 (DEFAULT_TABLE table=test-table [id=6f26c795e25f4d69824f04d36829368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:59:36.723763 12134 tablet_service.cc:1467] Processing CreateTablet for tablet c5cc826d0e0846d3a94156608c0570f5 (DEFAULT_TABLE table=test-table [id=6f26c795e25f4d69824f04d36829368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:59:36.724905 12000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cc826d0e0846d3a94156608c0570f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:59:36.725150 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cc826d0e0846d3a94156608c0570f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:59:36.724978 11857 tablet_service.cc:1467] Processing CreateTablet for tablet c5cc826d0e0846d3a94156608c0570f5 (DEFAULT_TABLE table=test-table [id=6f26c795e25f4d69824f04d36829368a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:59:36.726241 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5cc826d0e0846d3a94156608c0570f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:59:36.736840 12218 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Bootstrap starting.
I20260620 06:59:36.739164 12219 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Bootstrap starting.
I20260620 06:59:36.740536 12218 tablet_bootstrap.cc:654] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Neither blocks nor log segments found. Creating new log.
I20260620 06:59:36.741900 12218 log.cc:824] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Log is configured to *not* fsync() on all Append() calls
I20260620 06:59:36.743022 12219 tablet_bootstrap.cc:654] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Neither blocks nor log segments found. Creating new log.
I20260620 06:59:36.744206 12219 log.cc:824] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:59:36.745260 12218 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: No bootstrap required, opened a new log
I20260620 06:59:36.745517 12218 ts_tablet_manager.cc:1397] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:59:36.746654 12220 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Bootstrap starting.
I20260620 06:59:36.747013 12219 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: No bootstrap required, opened a new log
I20260620 06:59:36.747246 12219 ts_tablet_manager.cc:1397] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 06:59:36.749627 12220 tablet_bootstrap.cc:654] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Neither blocks nor log segments found. Creating new log.
I20260620 06:59:36.750494 12220 log.cc:824] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Log is configured to *not* fsync() on all Append() calls
I20260620 06:59:36.752970 12220 tablet_bootstrap.cc:492] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: No bootstrap required, opened a new log
I20260620 06:59:36.753208 12220 ts_tablet_manager.cc:1397] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 06:59:36.755664 12218 raft_consensus.cc:348] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.756115 12218 raft_consensus.cc:374] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:59:36.756260 12218 raft_consensus.cc:729] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19be3001db774c988f9bc72f39b402ee, State: Initialized, Role: FOLLOWER
I20260620 06:59:36.756862 12218 consensus_queue.cc:260] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.757957 12065 heartbeater.cc:503] Master 127.0.97.62:45645 was elected leader, sending a full tablet report...
I20260620 06:59:36.758610 12218 ts_tablet_manager.cc:1428] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260620 06:59:36.758951 12219 raft_consensus.cc:348] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.759336 12219 raft_consensus.cc:374] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:59:36.759431 12219 raft_consensus.cc:729] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4006044e439542caa6423e7bd79e89a2, State: Initialized, Role: FOLLOWER
I20260620 06:59:36.759943 12219 consensus_queue.cc:260] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.760897 12199 heartbeater.cc:503] Master 127.0.97.62:45645 was elected leader, sending a full tablet report...
I20260620 06:59:36.761477 12219 ts_tablet_manager.cc:1428] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260620 06:59:36.761616 12220 raft_consensus.cc:348] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.762010 12220 raft_consensus.cc:374] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:59:36.762107 12220 raft_consensus.cc:729] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7703b2381c6042d283f0f24a33c61614, State: Initialized, Role: FOLLOWER
I20260620 06:59:36.762709 12220 consensus_queue.cc:260] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
W20260620 06:59:36.763862 12066 tablet.cc:2369] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:59:36.764364 12220 ts_tablet_manager.cc:1428] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260620 06:59:36.880326 12200 tablet.cc:2369] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:59:36.898700 12225 raft_consensus.cc:482] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:59:36.898981 12225 raft_consensus.cc:504] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.900774 12225 leader_election.cc:288] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7703b2381c6042d283f0f24a33c61614 (127.0.97.1:45211), 19be3001db774c988f9bc72f39b402ee (127.0.97.2:40649)
I20260620 06:59:36.906991 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.901316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41349 (local address 127.0.97.2:40649)
0620 06:59:36.901488 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.901492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.901507 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:59:36.901763 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.901768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.901857 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.901965 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:59:36.902420 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.903396 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.905194 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.906027 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.906127 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.906196 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.906572 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.906698 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:59:36.907078 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.901166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51403 (local address 127.0.97.1:45211)
0620 06:59:36.901367 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.901373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.902035 (+ 662us) server_negotiation.cc:407] Connection header received
0620 06:59:36.902200 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.902206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.902282 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.902375 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:59:36.902959 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.903698 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.905443 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.906245 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.906372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.906465 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.906767 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.906896 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:59:36.907457 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.901063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40649 (local address 127.0.97.3:41349)
0620 06:59:36.901433 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.901451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.901584 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.901988 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.901991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.902019 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:59:36.902279 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.902292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.903539 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.903544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.905030 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.905041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.906182 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.906188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.906364 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.906401 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.906514 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.906664 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":22,"spinlock_wait_cycles":23424}
I20260620 06:59:36.907784 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.901244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45211 (local address 127.0.97.3:51403)
0620 06:59:36.901934 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 06:59:36.901955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:59:36.902064 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:59:36.902407 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:59:36.902411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:59:36.902434 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:59:36.902800 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.902811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.903836 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.903840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.905269 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:59:36.905282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.906432 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.906436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:59:36.906603 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.906634 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:59:36.906718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:59:36.906834 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":23,"thread_start_us":383,"threads_started":1}
I20260620 06:59:36.908530 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cc826d0e0846d3a94156608c0570f5" candidate_uuid: "4006044e439542caa6423e7bd79e89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19be3001db774c988f9bc72f39b402ee" is_pre_election: true
I20260620 06:59:36.908648 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cc826d0e0846d3a94156608c0570f5" candidate_uuid: "4006044e439542caa6423e7bd79e89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7703b2381c6042d283f0f24a33c61614" is_pre_election: true
I20260620 06:59:36.909024 12020 raft_consensus.cc:2454] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4006044e439542caa6423e7bd79e89a2 in term 0.
I20260620 06:59:36.909143 11877 raft_consensus.cc:2454] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4006044e439542caa6423e7bd79e89a2 in term 0.
I20260620 06:59:36.909670 12087 leader_election.cc:302] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19be3001db774c988f9bc72f39b402ee, 4006044e439542caa6423e7bd79e89a2, 7703b2381c6042d283f0f24a33c61614; no voters:
I20260620 06:59:36.910107 12225 raft_consensus.cc:2781] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:59:36.910262 12225 raft_consensus.cc:482] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:59:36.910352 12225 raft_consensus.cc:3037] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:59:36.912544 12225 raft_consensus.cc:504] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.913070 12225 leader_election.cc:288] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [CANDIDATE]: Term 1 election: Requested vote from peers 7703b2381c6042d283f0f24a33c61614 (127.0.97.1:45211), 19be3001db774c988f9bc72f39b402ee (127.0.97.2:40649)
I20260620 06:59:36.913506 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cc826d0e0846d3a94156608c0570f5" candidate_uuid: "4006044e439542caa6423e7bd79e89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7703b2381c6042d283f0f24a33c61614"
I20260620 06:59:36.913503 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5cc826d0e0846d3a94156608c0570f5" candidate_uuid: "4006044e439542caa6423e7bd79e89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19be3001db774c988f9bc72f39b402ee"
I20260620 06:59:36.913698 12020 raft_consensus.cc:3037] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:59:36.913698 11877 raft_consensus.cc:3037] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:59:36.916728 12020 raft_consensus.cc:2454] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4006044e439542caa6423e7bd79e89a2 in term 1.
I20260620 06:59:36.916728 11877 raft_consensus.cc:2454] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4006044e439542caa6423e7bd79e89a2 in term 1.
I20260620 06:59:36.917173 12088 leader_election.cc:302] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19be3001db774c988f9bc72f39b402ee, 4006044e439542caa6423e7bd79e89a2, 7703b2381c6042d283f0f24a33c61614; no voters:
I20260620 06:59:36.917481 12225 raft_consensus.cc:2781] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:59:36.917886 12225 raft_consensus.cc:686] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 1 LEADER]: Becoming Leader. State: Replica: 4006044e439542caa6423e7bd79e89a2, State: Running, Role: LEADER
I20260620 06:59:36.918368 12225 consensus_queue.cc:237] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } }
I20260620 06:59:36.923348 11709 catalog_manager.cc:5495] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4006044e439542caa6423e7bd79e89a2 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4006044e439542caa6423e7bd79e89a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4006044e439542caa6423e7bd79e89a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 } health_report { overall_health: UNKNOWN } } }
I20260620 06:59:36.952203 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.946712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.0.97.3:36847)
0620 06:59:36.946872 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.946877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:36.946971 (+ 94us) server_negotiation.cc:407] Connection header received
0620 06:59:36.947134 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:36.947137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:36.947222 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:36.947368 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:59:36.947785 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.948489 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.950079 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:36.950509 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:36.950639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:36.951121 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:59:36.951661 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:59:36.951772 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:36.951922 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 06:59:36.951987 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:59:36.960389 11877 raft_consensus.cc:1261] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 1 FOLLOWER]: Refusing update from remote peer 4006044e439542caa6423e7bd79e89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:59:36.960389 12020 raft_consensus.cc:1261] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 1 FOLLOWER]: Refusing update from remote peer 4006044e439542caa6423e7bd79e89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:59:36.961367 12228 consensus_queue.cc:1035] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7703b2381c6042d283f0f24a33c61614" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:59:36.961786 12225 consensus_queue.cc:1035] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19be3001db774c988f9bc72f39b402ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:59:36.967911 11923 tablet.cc:2369] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:59:36.973657 12231 mvcc.cc:205] Tried to move back new op lower bound from 7298821230419623936 to 7298821230260854784. Current Snapshot: MvccSnapshot[applied={T|T < 7298821230419623936}]
I20260620 06:59:36.989239 11709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:59:36.989480 11709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:59:36.992799 11709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf86bbbc87574ff5a90ed3cfebdffb6c: Sending DeleteTablet for 3 replicas of tablet c5cc826d0e0846d3a94156608c0570f5
I20260620 06:59:36.993914 12000 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cc826d0e0846d3a94156608c0570f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42038
I20260620 06:59:36.994163 12134 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cc826d0e0846d3a94156608c0570f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48608
I20260620 06:59:36.994843 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet c5cc826d0e0846d3a94156608c0570f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:59:36 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44402
I20260620 06:59:36.995383 12241 tablet_replica.cc:330] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: stopping tablet replica
I20260620 06:59:36.995832 12241 raft_consensus.cc:2229] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:59:36.996361 12241 raft_consensus.cc:2258] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:59:36.996554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 11789
I20260620 06:59:37.005218 12241 ts_tablet_manager.cc:1905] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:59:37.009219 12242 tablet_replica.cc:330] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: stopping tablet replica
I20260620 06:59:37.009809 12242 raft_consensus.cc:2229] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:59:37.010133 12243 tablet_replica.cc:330] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: stopping tablet replica
I20260620 06:59:37.010560 12242 raft_consensus.cc:2258] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:59:37.010703 12243 raft_consensus.cc:2229] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:59:37.011313 12243 raft_consensus.cc:2258] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:59:37.013535 12243 ts_tablet_manager.cc:1905] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:59:37.013542 12242 ts_tablet_manager.cc:1905] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:59:37.018190 12241 ts_tablet_manager.cc:1918] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:59:37.018390 12241 log.cc:1196] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/c5cc826d0e0846d3a94156608c0570f5
I20260620 06:59:37.018998 12241 ts_tablet_manager.cc:1939] T c5cc826d0e0846d3a94156608c0570f5 P 19be3001db774c988f9bc72f39b402ee: Deleting consensus metadata
I20260620 06:59:37.019976 12243 ts_tablet_manager.cc:1918] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:59:37.020385 12243 log.cc:1196] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/c5cc826d0e0846d3a94156608c0570f5
I20260620 06:59:37.020828 11693 catalog_manager.cc:4841] TS 19be3001db774c988f9bc72f39b402ee (127.0.97.2:40649): tablet c5cc826d0e0846d3a94156608c0570f5 (table test-table [id=6f26c795e25f4d69824f04d36829368a]) successfully deleted
I20260620 06:59:37.020926 12243 ts_tablet_manager.cc:1939] T c5cc826d0e0846d3a94156608c0570f5 P 7703b2381c6042d283f0f24a33c61614: Deleting consensus metadata
I20260620 06:59:37.022580 11694 catalog_manager.cc:4841] TS 7703b2381c6042d283f0f24a33c61614 (127.0.97.1:45211): tablet c5cc826d0e0846d3a94156608c0570f5 (table test-table [id=6f26c795e25f4d69824f04d36829368a]) successfully deleted
I20260620 06:59:37.025298 12242 ts_tablet_manager.cc:1918] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:59:37.025513 12242 log.cc:1196] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/c5cc826d0e0846d3a94156608c0570f5
I20260620 06:59:37.026043 12242 ts_tablet_manager.cc:1939] T c5cc826d0e0846d3a94156608c0570f5 P 4006044e439542caa6423e7bd79e89a2: Deleting consensus metadata
I20260620 06:59:37.027921 11694 catalog_manager.cc:4841] TS 4006044e439542caa6423e7bd79e89a2 (127.0.97.3:36847): tablet c5cc826d0e0846d3a94156608c0570f5 (table test-table [id=6f26c795e25f4d69824f04d36829368a]) successfully deleted
Jun 20 06:59:37 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 06:59:37.040197 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:36.997903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.0.97.1:45211)
0620 06:59:36.998086 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:36.998092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:37.013359 (+ 15267us) server_negotiation.cc:407] Connection header received
0620 06:59:37.013561 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:37.013566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:37.013645 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:37.013752 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:37.023103 (+ 9351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.024228 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.026216 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.026850 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.026943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:37.032613 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:37.032631 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:37.032650 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:37.032704 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:37.037136 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.037940 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.037951 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.037955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.038119 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.038490 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.038495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.038499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.039026 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:37.039321 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:37.039596 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 06:59:37.039816 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:59:37.040983 11917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:59:37.130239 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 11789
I20260620 06:59:37.144459 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 11928
Jun 20 06:59:37 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 06:59:37.161911 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:37.145225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.0.97.2:40649)
0620 06:59:37.145395 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:37.145400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:37.145418 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:59:37.145607 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:37.145611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:37.145691 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:37.145782 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:37.147345 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.148087 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.149933 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.150319 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.150413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:37.155490 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:37.155504 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:37.155521 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:37.155560 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:37.159053 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.159840 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.159851 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.159855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.160032 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.160428 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.160431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.160433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.160801 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:37.161014 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:37.161431 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 06:59:37.161590 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:59:37.162571 12060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:59:37.250104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 11928
I20260620 06:59:37.270114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 12071
Jun 20 06:59:37 dist-test-slave-mz20 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 06:59:37.288026 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:59:37.270864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.0.97.3:36847)
0620 06:59:37.271030 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:59:37.271035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:59:37.271056 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:59:37.271258 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:59:37.271261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:59:37.271367 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:59:37.271452 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:59:37.273162 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.273927 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.275807 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:59:37.276285 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:59:37.276405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:59:37.281768 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:59:37.281804 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:59:37.281821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:59:37.281866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:59:37.285030 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.285929 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.285938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.285940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.286096 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:59:37.286520 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:59:37.286523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:59:37.286525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:59:37.286936 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:59:37.287174 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 06:59:37.287559 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 06:59:37.287734 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:59:37.288741 12194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:59:37.407248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 12071
I20260620 06:59:37.422374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 11677
I20260620 06:59:37.423554 11739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 06:59:37.535758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 11677
I20260620 06:59:37.555970 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:59:37.891342 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:59:37.891577 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:59:38.227797 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:59:38.228 UTC [11263] LOG: received smart shutdown request
2026-06-20 06:59:38.231 UTC [11263] LOG: background worker "logical replication launcher" (PID 11272) exited with exit code 1
2026-06-20 06:59:38.232 UTC [11267] LOG: shutting down
2026-06-20 06:59:38.383 UTC [11263] LOG: database system is shut down
2026-06-20T06:59:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47446 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:59:38 dist-test-slave-mz20 krb5kdc[12257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:59:38 dist-test-slave-mz20 krb5kdc[12257](info): set up 2 sockets
Jun 20 06:59:38 dist-test-slave-mz20 krb5kdc[12257](info): commencing operation
krb5kdc: starting...
W20260620 06:59:39.527014 388 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 06:59:39 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:59:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:59:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 06:59:39.823480 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8UMGcc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/postgres -l logfile start
2026-06-20 06:59:42.192 UTC [12287] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:59:42.193 UTC [12287] LOG: listening on IPv4 address "127.0.97.20", port 50889
2026-06-20 06:59:42.196 UTC [12287] LOG: listening on Unix socket "/tmp/.s.PGSQL.50889"
2026-06-20 06:59:42.215 UTC [12290] LOG: database system was shut down at 2026-06-20 06:59:40 UTC
2026-06-20 06:59:42.218 UTC [12287] LOG: database system is ready to accept connections
I20260620 06:59:43.110355 388 mini_postgres.cc:96] Postgres bound to 50889
2026-06-20 06:59:43.126 UTC [12298] FATAL: database "slave" does not exist
127.0.97.20:50889 - accepting connections
I20260620 06:59:43.126711 388 mini_ranger.cc:161] Starting Ranger...
I20260620 06:59:43.155812 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:59:43.600906 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:59:43.601008 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin
2026-06-20 06:59:43,884 [I] DB FLAVOR :POSTGRES
2026-06-20 06:59:43,884 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:59:43,884 [I] Checking connection..
2026-06-20 06:59:43,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:59:44,126 [I] Checking connection passed.
2026-06-20 06:59:44,126 [I] --------- Verifying version history table ---------
2026-06-20 06:59:44,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:59:44,386 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:59:44,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:59:44,626 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:59:44,626 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:59:44,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:59:44.869 UTC [12362] WARNING: there is no transaction in progress
2026-06-20 06:59:44,875 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:59:44,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:59:45,108 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:59:45,108 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:59:45,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:59:45,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:59:45,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mz20.c.gcp-upstream.internal','N') ;"
2026-06-20 06:59:45,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:59:46,072 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:59:46,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:59:46,317 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:59:46,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:59:46,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:59:46,792 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:59:46,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:59:47.692 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.707 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.722 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.736 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.754 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.771 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.824 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.831 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.845 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.854 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.864 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.876 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.886 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.892 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.899 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:47.905 UTC [12470] WARNING: there is no transaction in progress
2026-06-20 06:59:48,517 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:59:48,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:59:48,753 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:59:48,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:59:49,008 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:59:49,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:59:49,263 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:59:49,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:59:49,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:59:49,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mz20.c.gcp-upstream.internal';"
2026-06-20 06:59:49,995 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:59:49,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:59:50,251 [I] DB_PATCHES have already been applied
I20260620 06:59:50.257486 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8UMGcc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:59:50.257575 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 06:59:50.270526 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:33967
Jun 20, 2026 6:59:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:59:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:59:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:59:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:59:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:59:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:59:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:59:51.303049 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.176s user 0.003s sys 0.038s
Jun 20, 2026 7:00:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:00:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:00:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:00:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:00:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 07:00:12.219388 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 07:00:12 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938812, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 07:00:12.373615 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 07:00:12.392915 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 07:00:13.268363 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 07:00:13.268507 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms
I20260620 07:00:13.268553 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:50889
2026-06-20 07:00:13,595 [I] DB FLAVOR :POSTGRES
2026-06-20 07:00:13,595 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 07:00:13,595 [I] Checking connection
2026-06-20 07:00:13,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 07:00:13,827 [I] connection success
2026-06-20 07:00:13,828 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 07:00:13,828 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 07:00:13,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 07:00:14,054 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 07:00:14,054 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 07:00:14,054 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 07:00:14,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50889/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 07:00:14,301 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 07:00:14.795825 388 mini_ranger_kms.cc:310] Created kms service
I20260620 07:00:14.987260 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 07:00:15.047534 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 07:00:15 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938812, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:15.457454 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 07:00:15.457778 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task8UMGcc/build/asan/bin/postgresql.jar:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8UMGcc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 07:00:15.457842 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 07:00:15.472951 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12650
I20260620 07:00:15.473178 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:57481
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:00:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 07:00:17.537405 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.164s user 0.081s sys 0.262s
I20260620 07:00:17.537624 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 07:00:17.537772 388 mini_ranger_kms.cc:401] 127.0.97.20:57481/kms/v1/keys
Jun 20 07:00:17 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781938817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 07:00:18 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 07:00:18 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938817, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 07:00:20 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:00:20.706593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39663
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39663
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:21.033054 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:21.033362 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:21.033483 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:21.040810 12701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:21.040934 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:21.040984 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:21.041060 12701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:21.041122 12701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:21.051282 12701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:57481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39663
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:21.052588 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:21.053934 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:21.065512 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:21.065461 12709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:21.065732 12701 server_base.cc:1034] running on GCE node
W20260620 07:00:21.065433 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:21.066766 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:21.068169 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:21.069365 12701 hybrid_clock.cc:648] HybridClock initialized: now 1781938821069319 us; error 32 us; skew 500 ppm
I20260620 07:00:21.069696 12701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:21 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:21.072829 12701 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:21.074293 12701 webserver.cc:460] Webserver started at http://127.0.97.62:36129/ using document root <none> and password file <none>
I20260620 07:00:21.074991 12701 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:21.075098 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:21.075371 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:00:21 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:21.124328 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "819df99890714e14ac7be6489267625a"
format_stamp: "Formatted at 2026-06-20 07:00:21 on dist-test-slave-mz20"
server_key: "e64b4851f6b5de814ca6079775c2f2be"
server_key_iv: "cb0899c9cf654aa6b2bff10c1eeaeada"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:21.125214 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "819df99890714e14ac7be6489267625a"
format_stamp: "Formatted at 2026-06-20 07:00:21 on dist-test-slave-mz20"
server_key: "e64b4851f6b5de814ca6079775c2f2be"
server_key_iv: "cb0899c9cf654aa6b2bff10c1eeaeada"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:21.130793 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:00:21.152912 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:21.154973 12701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 07:00:21.155153 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "819df99890714e14ac7be6489267625a"
format_stamp: "Formatted at 2026-06-20 07:00:21 on dist-test-slave-mz20"
server_key: "e64b4851f6b5de814ca6079775c2f2be"
server_key_iv: "cb0899c9cf654aa6b2bff10c1eeaeada"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:21.155372 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:21.176992 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:21.181111 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:21.181411 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:21.198779 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39663
I20260620 07:00:21.198830 12767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39663 every 8 connection(s)
I20260620 07:00:21.200122 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:21.201637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 12701
I20260620 07:00:21.201854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:00:21.202695 388 external_mini_cluster.cc:1428] Setting key cc61627bdc9ff4ab668c2dbd5fe8d894
I20260620 07:00:21.204967 12768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:00:21 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:21.216133 12768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Bootstrap starting.
I20260620 07:00:21.219481 12768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:21.220674 12768 log.cc:824] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:21.224052 12768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: No bootstrap required, opened a new log
I20260620 07:00:21.224310 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:21.204834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.0.97.62:39663)
0620 07:00:21.205658 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:21.205670 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:21.205712 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:00:21.207367 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:21.207391 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:21.207805 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:21.208219 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:21.209525 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:21.210949 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:21.212177 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:21.212919 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:21.213086 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:21.216985 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:21.216994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:21.217010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:21.217054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:21.219956 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:21.220796 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:21.220804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:21.220813 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:21.220981 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:21.221406 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:21.221411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:21.221414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:21.222187 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:21.222395 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:21.222953 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 07:00:21.223294 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":352,"threads_started":1}
I20260620 07:00:21.231666 12768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:21.232035 12768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:21.232136 12768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819df99890714e14ac7be6489267625a, State: Initialized, Role: FOLLOWER
I20260620 07:00:21.232697 12768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:21.232896 12768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:21.233026 12768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:21.233174 12768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:21.235373 12768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:21.235857 12768 leader_election.cc:302] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819df99890714e14ac7be6489267625a; no voters:
I20260620 07:00:21.236389 12768 leader_election.cc:288] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:00:21.236505 12773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:21.237149 12773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 LEADER]: Becoming Leader. State: Replica: 819df99890714e14ac7be6489267625a, State: Running, Role: LEADER
I20260620 07:00:21.237713 12773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:21.238374 12768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:21.240188 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819df99890714e14ac7be6489267625a. Latest consensus state: current_term: 1 leader_uuid: "819df99890714e14ac7be6489267625a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } } }
I20260620 07:00:21.240530 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:21.241317 12781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:21.241261 12774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819df99890714e14ac7be6489267625a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } } }
I20260620 07:00:21.241544 12774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:21.247082 12781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:21.255678 12768 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 07:00:21.269061 12781 catalog_manager.cc:1347] Generated new cluster ID: 0e155a4357a24659bef0a47417e8191a
I20260620 07:00:21.269243 12781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:21.286198 12781 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:00:21.288309 12781 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:21.302475 12781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Generated new TSK 0
I20260620 07:00:21.303745 12781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:00:22.054904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:22 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 07:00:22.487723 12813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:22.488143 12813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:22.488363 12813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:22.500240 12813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:22.500419 12813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:22.500669 12813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:00:22.516808 12813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:22.518584 12813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:22.520242 12813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:22.552611 12823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:22.555089 12824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:22.564536 12813 server_base.cc:1034] running on GCE node
W20260620 07:00:22.564888 12826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:22.565793 12813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:22.567114 12813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:22.568982 12813 hybrid_clock.cc:648] HybridClock initialized: now 1781938822568861 us; error 110 us; skew 500 ppm
I20260620 07:00:22.569401 12813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:22 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:22.573688 12813 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:00:22.575745 12813 webserver.cc:460] Webserver started at http://127.0.97.1:45193/ using document root <none> and password file <none>
I20260620 07:00:22.576661 12813 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:22.576792 12813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:22.577131 12813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:00:22 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:22.596561 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "9d6003d59783495a9bcbacf2c7b27294"
format_stamp: "Formatted at 2026-06-20 07:00:22 on dist-test-slave-mz20"
server_key: "d979ba39c09aae685c958d60db5ab4b9"
server_key_iv: "639473b2a8cc344645b2a018e13db937"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:22.597785 12813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9d6003d59783495a9bcbacf2c7b27294"
format_stamp: "Formatted at 2026-06-20 07:00:22 on dist-test-slave-mz20"
server_key: "d979ba39c09aae685c958d60db5ab4b9"
server_key_iv: "639473b2a8cc344645b2a018e13db937"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:22.613808 12813 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.007s sys 0.000s
I20260620 07:00:22.638517 12833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:22.640645 12813 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260620 07:00:22.640951 12813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "9d6003d59783495a9bcbacf2c7b27294"
format_stamp: "Formatted at 2026-06-20 07:00:22 on dist-test-slave-mz20"
server_key: "d979ba39c09aae685c958d60db5ab4b9"
server_key_iv: "639473b2a8cc344645b2a018e13db937"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:22.641315 12813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:22.689440 12813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:22.693902 12813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:22.694298 12813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:22.695387 12813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:22.696883 12813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:22.696961 12813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:22.697080 12813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:22.697136 12813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:22.759276 12813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42741
I20260620 07:00:22.759286 12945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42741 every 8 connection(s)
I20260620 07:00:22.760773 12813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:00:22.768834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 12813
I20260620 07:00:22.769042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:00:22.769865 388 external_mini_cluster.cc:1428] Setting key f3539013eab0844276bfa74af1709e93
Jun 20 07:00:22 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:22.797811 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:22.763887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49517 (local address 127.0.97.62:39663)
0620 07:00:22.774926 (+ 11039us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:22.774932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:22.778451 (+ 3519us) server_negotiation.cc:407] Connection header received
0620 07:00:22.778526 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:22.778532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:22.778658 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:22.778768 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:22.781051 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:22.782287 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:22.784929 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:22.785574 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:22.785676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:22.790206 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:22.790214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:22.790219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:22.790260 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:22.793851 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:22.794880 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:22.794885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:22.794889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:22.795043 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:22.795618 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:22.795623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:22.795626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:22.796001 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:22.796141 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:22.797169 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0620 07:00:22.797420 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10901,"thread_start_us":10774,"threads_started":1}
I20260620 07:00:22.800829 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:22.764502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.1:49517)
0620 07:00:22.775409 (+ 10907us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:22.775476 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:22.777948 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:22.779277 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:22.779292 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:22.779948 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:22.780822 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:22.780849 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:22.782502 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:22.782507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:22.784738 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:22.784764 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:22.786037 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:22.786045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:22.786369 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:22.787137 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:22.787169 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:22.789922 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:22.794056 (+ 4134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:22.794066 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:22.794085 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:22.794725 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:22.795215 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:22.795220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:22.795224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:22.795482 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:22.796247 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:22.796255 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:22.796717 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 07:00:22.798607 (+ 1890us) client_negotiation.cc:240] Negotiation successful
0620 07:00:22.799185 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10543,"thread_start_us":16172,"threads_started":1}
I20260620 07:00:22.803359 12946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:22.803720 12946 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:22.804625 12946 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:22.807772 12733 ts_manager.cc:194] Registered new tserver with Master: 9d6003d59783495a9bcbacf2c7b27294 (127.0.97.1:42741)
I20260620 07:00:22.809865 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49517
Jun 20 07:00:22 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:00:22.863265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:23 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 07:00:23.362495 12952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:23.363029 12952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:23.363209 12952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:23.374588 12952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:23.374811 12952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:23.375041 12952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:00:23.390014 12952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:23.391918 12952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:23.393613 12952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:23.429227 12960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:23.433293 12959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:23.441039 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:23.446257 12952 server_base.cc:1034] running on GCE node
I20260620 07:00:23.447436 12952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:23.467486 12952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:23.471043 12952 hybrid_clock.cc:648] HybridClock initialized: now 1781938823470926 us; error 121 us; skew 500 ppm
I20260620 07:00:23.471586 12952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:23 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:23.477594 12952 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:00:23.486619 12952 webserver.cc:460] Webserver started at http://127.0.97.2:33001/ using document root <none> and password file <none>
I20260620 07:00:23.487762 12952 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:23.487993 12952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:23.488385 12952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:00:23 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:23.513327 12952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19"
format_stamp: "Formatted at 2026-06-20 07:00:23 on dist-test-slave-mz20"
server_key: "fe90e85c213da72b7788a426206add4c"
server_key_iv: "0bfdad20f3c35707bb5d937a0dba78a7"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:23.514493 12952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19"
format_stamp: "Formatted at 2026-06-20 07:00:23 on dist-test-slave-mz20"
server_key: "fe90e85c213da72b7788a426206add4c"
server_key_iv: "0bfdad20f3c35707bb5d937a0dba78a7"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:23.521987 12952 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 07:00:23.555152 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:23.557220 12952 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.010s
I20260620 07:00:23.557504 12952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19"
format_stamp: "Formatted at 2026-06-20 07:00:23 on dist-test-slave-mz20"
server_key: "fe90e85c213da72b7788a426206add4c"
server_key_iv: "0bfdad20f3c35707bb5d937a0dba78a7"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:23.557832 12952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:23.589905 12952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:23.595052 12952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:23.595430 12952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:23.596603 12952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:23.598353 12952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:23.598455 12952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:23.598635 12952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:23.598706 12952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:23.642184 12952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45685
I20260620 07:00:23.642234 13088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45685 every 8 connection(s)
I20260620 07:00:23.643930 12952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:00:23.650975 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 12952
I20260620 07:00:23.651154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:00:23.651784 388 external_mini_cluster.cc:1428] Setting key d4bac2760b178d015da28e0c0a40f766
Jun 20 07:00:23 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:23.674307 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:23.646904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47577 (local address 127.0.97.62:39663)
0620 07:00:23.652951 (+ 6047us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:23.652957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:23.652980 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:00:23.654358 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:23.654365 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:23.654453 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:23.654571 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:23.657624 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:23.658759 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:23.661221 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:23.661859 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:23.661963 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:23.666799 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:23.666806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:23.666811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:23.666853 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:23.670426 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:23.671489 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:23.671495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:23.671498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:23.671652 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:23.672186 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:23.672192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:23.672195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:23.672580 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:23.672710 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:23.673692 (+ 982us) server_negotiation.cc:299] Negotiation successful
0620 07:00:23.673935 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5896,"thread_start_us":15773,"threads_started":1}
I20260620 07:00:23.676856 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:23.647739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.2:47577)
0620 07:00:23.648588 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:23.648643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:23.653900 (+ 5257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:23.655976 (+ 2076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:23.655988 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:23.656607 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:23.657429 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:23.657454 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:23.658937 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:23.658942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:23.661047 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:23.661070 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:23.662280 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:23.662298 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:23.662615 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:23.663569 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:23.663603 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:23.666507 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:23.670666 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:23.670676 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:23.670696 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:23.671329 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:23.671797 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:23.671801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:23.671805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:23.672061 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:23.672799 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:23.672808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:23.673248 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 07:00:23.674821 (+ 1573us) client_negotiation.cc:240] Negotiation successful
0620 07:00:23.675395 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":316,"threads_started":1}
I20260620 07:00:23.678741 13089 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:23.679051 13089 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:23.679847 13089 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 07:00:23.681573 12733 ts_manager.cc:194] Registered new tserver with Master: bd32429e6b3e4e0ab6be3d1f55b7ee19 (127.0.97.2:45685)
I20260620 07:00:23.682667 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:00:23.718626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:23.814109 12946 heartbeater.cc:503] Master 127.0.97.62:39663 was elected leader, sending a full tablet report...
W20260620 07:00:24.098278 13095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:24.098589 13095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:24.098726 13095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:24.105940 13095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:24.106055 13095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:24.106192 13095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:00:24.116609 13095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:24.117909 13095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:24.119257 13095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:24.133445 13101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:24.133554 13100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:24.134140 13103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:24.135620 13095 server_base.cc:1034] running on GCE node
I20260620 07:00:24.136226 13095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:24.137208 13095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:24.138415 13095 hybrid_clock.cc:648] HybridClock initialized: now 1781938824138341 us; error 55 us; skew 500 ppm
I20260620 07:00:24.138769 13095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:24.142138 13095 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:00:24.143726 13095 webserver.cc:460] Webserver started at http://127.0.97.3:39177/ using document root <none> and password file <none>
I20260620 07:00:24.144527 13095 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:24.144647 13095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:24.144943 13095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:24.162266 13095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "71dd9102b2e14c0db0102294ef8d3025"
format_stamp: "Formatted at 2026-06-20 07:00:24 on dist-test-slave-mz20"
server_key: "ff732b7ad30e3c830d800912ef00dd6c"
server_key_iv: "e9c9a6d09405bae4c8de170b2ceb5fab"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:24.163292 13095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "71dd9102b2e14c0db0102294ef8d3025"
format_stamp: "Formatted at 2026-06-20 07:00:24 on dist-test-slave-mz20"
server_key: "ff732b7ad30e3c830d800912ef00dd6c"
server_key_iv: "e9c9a6d09405bae4c8de170b2ceb5fab"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:24.168936 13095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 07:00:24.187222 13110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:24.189143 13095 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 07:00:24.189330 13095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "71dd9102b2e14c0db0102294ef8d3025"
format_stamp: "Formatted at 2026-06-20 07:00:24 on dist-test-slave-mz20"
server_key: "ff732b7ad30e3c830d800912ef00dd6c"
server_key_iv: "e9c9a6d09405bae4c8de170b2ceb5fab"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:24.189579 13095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:24.219111 13095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:24.223290 13095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:24.223577 13095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:24.224606 13095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:24.226086 13095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:24.226162 13095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:24.226259 13095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:24.226310 13095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:24.256752 13095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40699
I20260620 07:00:24.256785 13222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40699 every 8 connection(s)
I20260620 07:00:24.257956 13095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:00:24.262059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13095
I20260620 07:00:24.262235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:00:24.262743 388 external_mini_cluster.cc:1428] Setting key d5590150f92416a927aa2338c52af746
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:24.277257 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:24.260597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44345 (local address 127.0.97.62:39663)
0620 07:00:24.261038 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:24.261042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:24.261867 (+ 825us) server_negotiation.cc:407] Connection header received
0620 07:00:24.263835 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:24.263862 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:24.263937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:24.264057 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:24.265667 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.266477 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.268050 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.268495 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.268585 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:24.271807 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:24.271811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:24.271815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:24.271849 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:24.274364 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.275064 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.275067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.275070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.275198 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.275558 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.275560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.275562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.275822 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:24.275936 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:24.276760 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 07:00:24.276995 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
I20260620 07:00:24.278405 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:24.260980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.3:44345)
0620 07:00:24.261705 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:24.261749 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:24.263572 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:24.264309 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:24.264322 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:24.264855 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:24.265462 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:24.265486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.266657 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.266662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:24.267915 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:24.267935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.268673 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.268678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:24.268876 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:24.269536 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:24.269561 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:24.271581 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:24.274505 (+ 2924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:24.274512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:24.274525 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:24.274962 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:24.275276 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:24.275279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:24.275281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:24.275471 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:24.275942 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:24.275949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:24.276328 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 07:00:24.276661 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 07:00:24.277133 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":294,"threads_started":1}
I20260620 07:00:24.279673 13223 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:24.279940 13223 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:24.280622 13223 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:24.281956 12733 ts_manager.cc:194] Registered new tserver with Master: 71dd9102b2e14c0db0102294ef8d3025 (127.0.97.3:40699)
I20260620 07:00:24.282791 12733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44345
I20260620 07:00:24.290307 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:00:24.291957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 12813
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:00:24.308998 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:24.292948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.0.97.1:42741)
0620 07:00:24.293535 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:24.293543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:24.293568 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:00:24.293624 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:24.293630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:24.293840 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:24.293974 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:24.295497 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.296592 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.298052 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.298705 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.298840 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:24.303085 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:24.303100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:24.303116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:24.303160 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:24.306045 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.306910 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.306918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.306920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.307100 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.307467 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.307470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.307472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.307823 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:24.308075 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:24.308398 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 07:00:24.308625 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":277,"threads_started":1}
I20260620 07:00:24.310243 12941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:24.393460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 12813
I20260620 07:00:24.405779 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 12952
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:00:24.422292 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:24.406722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.0.97.2:45685)
0620 07:00:24.407249 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:24.407257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:24.407282 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:00:24.407339 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:24.407345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:24.407608 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:24.407739 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:24.408980 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.410056 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.411524 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.412112 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.412223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:24.416639 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:24.416653 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:24.416670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:24.416706 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:24.419411 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.420210 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.420218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.420220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.420358 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.420721 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.420725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.420727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.421083 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:24.421376 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:24.421689 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 07:00:24.421947 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":280,"threads_started":1}
I20260620 07:00:24.423666 13084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:24.510639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 12952
I20260620 07:00:24.523056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13095
Jun 20 07:00:24 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:00:24.539163 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:24.523964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.3:40699)
0620 07:00:24.524505 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:24.524512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:24.524541 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:00:24.524598 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:24.524604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:24.524818 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:24.524987 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:24.526324 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.527568 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.528905 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:24.529474 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:24.529592 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:24.533941 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:24.533976 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:24.533992 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:24.534031 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:24.536603 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.537247 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.537254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.537256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.537377 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:24.537711 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:24.537716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:24.537718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:24.538048 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:24.538304 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:24.538651 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 07:00:24.538855 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":239,"threads_started":1}
I20260620 07:00:24.540524 13218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:24.622637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13095
I20260620 07:00:24.635038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 12701
I20260620 07:00:24.636178 12763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:24.740837 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 12701
I20260620 07:00:24.758242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39663
--webserver_interface=127.0.97.62
--webserver_port=36129
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39663
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:25.067837 13235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:25.068154 13235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:25.068280 13235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:25.075539 13235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:25.075675 13235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:25.075750 13235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:25.075829 13235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:25.075891 13235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:25.086377 13235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39663
--ranger_config_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:25.087700 13235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:25.089039 13235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:25.100507 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:25.100544 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:25.100477 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:25.101513 13235 server_base.cc:1034] running on GCE node
I20260620 07:00:25.102288 13235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:25.103753 13235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:25.104998 13235 hybrid_clock.cc:648] HybridClock initialized: now 1781938825104921 us; error 46 us; skew 500 ppm
I20260620 07:00:25.105353 13235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:25 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:25.108532 13235 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:25.109997 13235 webserver.cc:460] Webserver started at http://127.0.97.62:36129/ using document root <none> and password file <none>
I20260620 07:00:25.110678 13235 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:25.110785 13235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:00:25 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:25.155964 13235 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:00:25.159333 13251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:25.160764 13235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:00:25.160920 13235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "819df99890714e14ac7be6489267625a"
format_stamp: "Formatted at 2026-06-20 07:00:21 on dist-test-slave-mz20"
server_key: "e64b4851f6b5de814ca6079775c2f2be"
server_key_iv: "cb0899c9cf654aa6b2bff10c1eeaeada"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:25.161868 13235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:25.177150 13235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:25.181416 13235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:25.181677 13235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:25.200085 13235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39663
I20260620 07:00:25.200105 13302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39663 every 8 connection(s)
I20260620 07:00:25.201541 13235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:25.208227 13303 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:00:25.209194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13235
I20260620 07:00:25.209965 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42741
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39663
--webserver_port=45193
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:25.211573 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Bootstrap starting.
I20260620 07:00:25.229593 13303 log.cc:824] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:25.238354 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:00:25.238895 13303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Bootstrap complete.
I20260620 07:00:25.247375 13303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:25.248214 13303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819df99890714e14ac7be6489267625a, State: Initialized, Role: FOLLOWER
I20260620 07:00:25.248772 13303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:25.248978 13303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:25.249086 13303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:25.249233 13303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:00:25.253033 13303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:25.253489 13303 leader_election.cc:302] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819df99890714e14ac7be6489267625a; no voters:
I20260620 07:00:25.254045 13303 leader_election.cc:288] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:00:25.254196 13307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:00:25.256233 13303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:25.256188 13307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [term 2 LEADER]: Becoming Leader. State: Replica: 819df99890714e14ac7be6489267625a, State: Running, Role: LEADER
I20260620 07:00:25.256699 13307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } }
I20260620 07:00:25.258872 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "819df99890714e14ac7be6489267625a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } } }
I20260620 07:00:25.259656 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:25.266036 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819df99890714e14ac7be6489267625a. Latest consensus state: current_term: 2 leader_uuid: "819df99890714e14ac7be6489267625a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819df99890714e14ac7be6489267625a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39663 } } }
I20260620 07:00:25.267341 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:25.268632 13312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:25.274255 13312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:25.275640 13312 catalog_manager.cc:1259] Loaded cluster ID: 0e155a4357a24659bef0a47417e8191a
I20260620 07:00:25.275740 13312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:25.279659 13312 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:25.281026 13312 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Loaded TSK: 0
I20260620 07:00:25.282574 13312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:00:25.285871 13303 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 07:00:25.649395 13305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:25.649947 13305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:25.650151 13305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:25.662111 13305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:25.662339 13305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:25.662559 13305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:00:25.678135 13305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=45193
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:25.680048 13305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:25.681847 13305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:25.730177 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:25.731043 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:25.732613 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:25.732637 13305 server_base.cc:1034] running on GCE node
I20260620 07:00:25.734019 13305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:25.736100 13305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:25.739794 13305 hybrid_clock.cc:648] HybridClock initialized: now 1781938825739725 us; error 63 us; skew 500 ppm
I20260620 07:00:25.740208 13305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:25 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:25.745450 13305 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:00:25.762981 13305 webserver.cc:460] Webserver started at http://127.0.97.1:45193/ using document root <none> and password file <none>
I20260620 07:00:25.764086 13305 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:25.764286 13305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:00:25 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:25.818693 13305 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.008s sys 0.011s
I20260620 07:00:25.826531 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:25.828537 13305 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260620 07:00:25.828727 13305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "9d6003d59783495a9bcbacf2c7b27294"
format_stamp: "Formatted at 2026-06-20 07:00:22 on dist-test-slave-mz20"
server_key: "d979ba39c09aae685c958d60db5ab4b9"
server_key_iv: "639473b2a8cc344645b2a018e13db937"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:25.829788 13305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:25.861604 13305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:25.867020 13305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:25.867341 13305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:25.868505 13305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:25.870023 13305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:25.870131 13305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:25.870254 13305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:25.870334 13305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:25.918509 13305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42741
I20260620 07:00:25.918527 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42741 every 8 connection(s)
I20260620 07:00:25.920442 13305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:00:25.927065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13305
I20260620 07:00:25.927845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45685
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39663
--webserver_port=33001
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:25 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:25.974414 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:25.928800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.1:55667)
0620 07:00:25.937924 (+ 9124us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:25.937971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:25.940318 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:25.944525 (+ 4207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:25.944538 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:25.945149 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:25.945945 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:25.945972 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:25.948496 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:25.948502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:25.950475 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:25.950494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:25.952815 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:25.952824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:25.953128 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:25.957561 (+ 4433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:25.957608 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:25.960676 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:25.968867 (+ 8191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:25.968920 (+ 53us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:25.968940 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:25.969670 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:25.970326 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:25.970332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:25.970336 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:25.970637 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:25.971813 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:25.971822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:25.972219 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 07:00:25.972519 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 07:00:25.973110 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8829,"thread_start_us":9720,"threads_started":1}
I20260620 07:00:25.977139 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:25.928729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55667 (local address 127.0.97.62:39663)
0620 07:00:25.941650 (+ 12921us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:25.941671 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:25.941732 (+ 61us) server_negotiation.cc:407] Connection header received
0620 07:00:25.942163 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:25.942184 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:25.942736 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:25.943173 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:25.946235 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:25.948275 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:25.950757 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:25.952121 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:25.952346 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:25.961062 (+ 8716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:25.961195 (+ 133us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:25.961244 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:25.961308 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:25.968627 (+ 7319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:25.969916 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:25.969926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:25.969936 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:25.970147 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:25.970794 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:25.970799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:25.970803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:25.971655 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:25.975286 (+ 3631us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:25.975559 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 07:00:25.975979 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12644,"thread_start_us":12429,"threads_started":1}
I20260620 07:00:25.980331 13465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:25.980697 13465 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:25.981590 13465 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
W20260620 07:00:25.984669 13465 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39663 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:00:25.984895 13465 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:25.985174 13465 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:25.986297 13465 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:25.986582 13465 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
W20260620 07:00:25.987584 13465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:00:26.319623 13469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:26.319916 13469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:26.320046 13469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:26.328732 13469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:26.328863 13469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:26.329012 13469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:00:26.340291 13469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33001
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:26.341626 13469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:26.342995 13469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:26.357784 13484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:26.357980 13483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:26.358505 13469 server_base.cc:1034] running on GCE node
W20260620 07:00:26.358803 13486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:26.359596 13469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:26.360833 13469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:26.362089 13469 hybrid_clock.cc:648] HybridClock initialized: now 1781938826362009 us; error 65 us; skew 500 ppm
I20260620 07:00:26.362474 13469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:26 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:26.366664 13469 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:00:26.399653 13469 webserver.cc:460] Webserver started at http://127.0.97.2:33001/ using document root <none> and password file <none>
I20260620 07:00:26.400837 13469 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:26.401021 13469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:00:26 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:26.443791 13469 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.008s sys 0.004s
I20260620 07:00:26.447715 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:26.449568 13469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:00:26.449834 13469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19"
format_stamp: "Formatted at 2026-06-20 07:00:23 on dist-test-slave-mz20"
server_key: "fe90e85c213da72b7788a426206add4c"
server_key_iv: "0bfdad20f3c35707bb5d937a0dba78a7"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:26.451325 13469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:26.475076 13469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:26.481135 13469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:26.481451 13469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:26.483372 13469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:26.485194 13469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:26.485303 13469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:26.485414 13469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:26.485488 13469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:00:26 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:26.557001 13469 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45685
I20260620 07:00:26.557054 13608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45685 every 8 connection(s)
I20260620 07:00:26.558754 13469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:00:26.569031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13469
I20260620 07:00:26.569794 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:57481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40699
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39663
--webserver_port=39177
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:26 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:26.586751 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:26.561356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45213 (local address 127.0.97.62:39663)
0620 07:00:26.564224 (+ 2868us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:26.564231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:26.565020 (+ 789us) server_negotiation.cc:407] Connection header received
0620 07:00:26.566993 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:26.566999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:26.567107 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:26.567244 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:26.568811 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:26.571475 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:26.573381 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:26.573819 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:26.573940 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:26.578975 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:26.578982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:26.578987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:26.579035 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:26.582619 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:26.583800 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:26.583807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:26.583811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:26.583980 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:26.584551 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:26.584556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:26.584560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:26.584933 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:26.585078 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:26.586088 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0620 07:00:26.586343 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2705,"thread_start_us":2559,"threads_started":1}
I20260620 07:00:26.589190 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:26.561811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.2:45213)
0620 07:00:26.564787 (+ 2976us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:26.564848 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:26.566685 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:26.567478 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:26.567488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:26.567950 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:26.568596 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:26.568615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:26.571761 (+ 3146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:26.571767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:26.573184 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:26.573200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:26.574079 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:26.574087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:26.574327 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:26.575049 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:26.575074 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:26.578615 (+ 3541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:26.582863 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:26.582876 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:26.582897 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:26.583605 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:26.584122 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:26.584127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:26.584131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:26.584410 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:26.585153 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:26.585162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:26.585613 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 07:00:26.587118 (+ 1505us) client_negotiation.cc:240] Negotiation successful
0620 07:00:26.587745 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":2509,"threads_started":1}
I20260620 07:00:26.591094 13609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:26.591466 13609 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:26.592440 13609 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:26.595122 13268 ts_manager.cc:194] Registered new tserver with Master: bd32429e6b3e4e0ab6be3d1f55b7ee19 (127.0.97.2:45685)
I20260620 07:00:26.597471 13268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45213
Jun 20 07:00:26 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:26.987725 13465 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:26.988178 13465 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:26.989436 13268 ts_manager.cc:194] Registered new tserver with Master: 9d6003d59783495a9bcbacf2c7b27294 (127.0.97.1:42741)
I20260620 07:00:26.990617 13268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55667
W20260620 07:00:27.009879 13613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:27.010298 13613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:27.010457 13613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:27.021776 13613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:27.021922 13613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:27.022099 13613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:00:27.036532 13613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:57481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39177
--tserver_master_addrs=127.0.97.62:39663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:27.038345 13613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:27.040138 13613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:27.072268 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:27.073804 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:27.078733 13613 server_base.cc:1034] running on GCE node
W20260620 07:00:27.079021 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:27.079887 13613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:27.081280 13613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:27.082602 13613 hybrid_clock.cc:648] HybridClock initialized: now 1781938827082527 us; error 61 us; skew 500 ppm
I20260620 07:00:27.083002 13613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:27 dist-test-slave-mz20 krb5kdc[12257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:27.087170 13613 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:00:27.091387 13613 webserver.cc:460] Webserver started at http://127.0.97.3:39177/ using document root <none> and password file <none>
I20260620 07:00:27.092388 13613 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:27.092547 13613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 07:00:27 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:27.114894 13613 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:00:27.118630 13630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:27.120499 13613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:00:27.120680 13613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "71dd9102b2e14c0db0102294ef8d3025"
format_stamp: "Formatted at 2026-06-20 07:00:24 on dist-test-slave-mz20"
server_key: "ff732b7ad30e3c830d800912ef00dd6c"
server_key_iv: "e9c9a6d09405bae4c8de170b2ceb5fab"
server_key_version: "kuduclusterkey@0"
I20260620 07:00:27.122064 13613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:27.156414 13613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:27.161651 13613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:27.161942 13613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:27.163162 13613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:27.164976 13613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:27.165081 13613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:27.165230 13613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:27.165315 13613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 07:00:27 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 07:00:27.411348 13613 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40699
I20260620 07:00:27.411448 13743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40699 every 8 connection(s)
I20260620 07:00:27.413126 13613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:00:27.419406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13613
I20260620 07:00:27.453086 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.420872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.0.97.62:39663)
0620 07:00:27.434382 (+ 13510us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.434389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.435050 (+ 661us) server_negotiation.cc:407] Connection header received
0620 07:00:27.435800 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.435806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.435934 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.436041 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:27.438123 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.439242 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.441205 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.441828 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.441932 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.445460 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:27.445467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:27.445472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:27.445536 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:27.449163 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.450055 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.450060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.450064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.450214 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.450663 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.450667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.450671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.451098 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:27.451337 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.452487 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.452702 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13359,"thread_start_us":13259,"threads_started":1}
Jun 20 07:00:27 dist-test-slave-mz20 krb5kdc[12257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:27.487356 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.436002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44965 (local address 127.0.97.62:39663)
0620 07:00:27.446712 (+ 10710us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.446718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.459849 (+ 13131us) server_negotiation.cc:407] Connection header received
0620 07:00:27.461826 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.461840 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.461938 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.462041 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:27.464298 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.465170 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.467731 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.468800 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.468982 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.473450 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:27.473458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:27.473463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:27.473532 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:27.483899 (+ 10367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.484937 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.484943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.484947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.485077 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.485569 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.485572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.485575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.485860 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:27.485995 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.486753 (+ 758us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.486987 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10548,"thread_start_us":10705,"threads_started":1}
I20260620 07:00:27.488451 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.435546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39663 (local address 127.0.97.3:44965)
0620 07:00:27.459649 (+ 24103us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.459705 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.461529 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.462608 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.462623 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.463248 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:27.464069 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.464096 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.465335 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.465341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.467569 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.467584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.469109 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.469116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.469410 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.470179 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:27.470212 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:27.473033 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:27.484080 (+ 11047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:27.484091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:27.484109 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:27.484792 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:27.485197 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:27.485202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:27.485206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:27.485439 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:27.485996 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:27.486005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:27.486367 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.486654 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.487205 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23777,"thread_start_us":23536,"threads_started":1}
I20260620 07:00:27.489887 13744 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39663
I20260620 07:00:27.490224 13744 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:27.491106 13744 heartbeater.cc:511] Master 127.0.97.62:39663 requested a full tablet report, sending...
I20260620 07:00:27.492627 13268 ts_manager.cc:194] Registered new tserver with Master: 71dd9102b2e14c0db0102294ef8d3025 (127.0.97.3:40699)
I20260620 07:00:27.493438 13268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44965
I20260620 07:00:27.496856 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:00:27.514739 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.499856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.97.62:39663)
0620 07:00:27.500061 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.500068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.500475 (+ 407us) server_negotiation.cc:407] Connection header received
0620 07:00:27.500610 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.500614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.500678 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.500760 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:27.502725 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.503466 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.505527 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.505928 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.506034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.509210 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:27.509214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:27.509218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:27.509259 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:27.511976 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.512776 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.512781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.512783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.512964 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:27.513363 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:27.513367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:27.513369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:27.513651 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:27.513835 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.514229 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.514418 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:00:27.519624 13268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:00:27.527065 13268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:00:27.570041 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.542729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45685 (local address 127.0.0.1:38412)
0620 07:00:27.557046 (+ 14317us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.557303 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.557563 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.558695 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.558704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.558741 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:27.559201 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.559216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.564336 (+ 5120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.564342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.566763 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.566776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.568802 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.568871 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.569141 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.569204 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.569336 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.569547 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14176,"mutex_wait_us":2,"thread_start_us":17117,"threads_started":1}
I20260620 07:00:27.570631 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.542729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40699 (local address 127.0.0.1:54202)
0620 07:00:27.559647 (+ 16918us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.559665 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.559782 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.562000 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.562004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.562032 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:27.562409 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.562422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.565237 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.565243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.567635 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.567650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.569777 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.569783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.570029 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.570062 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.570137 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.570269 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16799,"mutex_wait_us":741,"thread_start_us":8600,"threads_started":1}
I20260620 07:00:27.571007 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.543243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.0.97.2:45685)
0620 07:00:27.551821 (+ 8578us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.551833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.557426 (+ 5593us) server_negotiation.cc:407] Connection header received
0620 07:00:27.557769 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.557779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.558083 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.558277 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:27.559479 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.564153 (+ 4674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.567042 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.568212 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.568381 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.568629 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.570432 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.570659 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8382,"thread_start_us":8249,"threads_started":1}
I20260620 07:00:27.571602 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.543043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.97.3:40699)
0620 07:00:27.555256 (+ 12213us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.555266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.559758 (+ 4492us) server_negotiation.cc:407] Connection header received
0620 07:00:27.561172 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.561182 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.561460 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.561640 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:27.562662 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.564664 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.567828 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.569256 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.569386 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.569601 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.571091 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.571257 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12041,"thread_start_us":11511,"threads_started":1}
I20260620 07:00:27.572639 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.542229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60756)
0620 07:00:27.552437 (+ 10208us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.552480 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.552689 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.565237 (+ 12548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.565242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.565272 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:27.565887 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.565902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.567993 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.567999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.569830 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.569841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.571813 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.571821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.572057 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.572079 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.572158 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.572286 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9922,"mutex_wait_us":26,"thread_start_us":14858,"threads_started":1}
I20260620 07:00:27.575273 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.542384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.0.97.1:42741)
0620 07:00:27.564575 (+ 22191us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.564583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.564616 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:00:27.564711 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.564718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.564997 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.566085 (+ 1088us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:27.566326 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.567857 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.570054 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.571507 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.571617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.571861 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.572225 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.574954 (+ 2729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22004,"thread_start_us":22541,"threads_started":1}
I20260620 07:00:27.574893 13679 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7738aebe84b70a44e6281e87a2789 (DEFAULT_TABLE table=test-table [id=4921937fd2414751bf44655fc28c5f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:27.574887 13544 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7738aebe84b70a44e6281e87a2789 (DEFAULT_TABLE table=test-table [id=4921937fd2414751bf44655fc28c5f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:27.576287 13679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7738aebe84b70a44e6281e87a2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:27.576346 13544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7738aebe84b70a44e6281e87a2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:27.579358 13400 tablet_service.cc:1467] Processing CreateTablet for tablet a3f7738aebe84b70a44e6281e87a2789 (DEFAULT_TABLE table=test-table [id=4921937fd2414751bf44655fc28c5f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:27.580868 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3f7738aebe84b70a44e6281e87a2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:27.588531 13772 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Bootstrap starting.
I20260620 07:00:27.591533 13772 tablet_bootstrap.cc:654] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:27.592756 13773 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Bootstrap starting.
I20260620 07:00:27.592767 13772 log.cc:824] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:27.596035 13775 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Bootstrap starting.
I20260620 07:00:27.596807 13773 tablet_bootstrap.cc:654] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:27.598075 13773 log.cc:824] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:27.599676 13775 tablet_bootstrap.cc:654] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:27.599680 13772 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: No bootstrap required, opened a new log
I20260620 07:00:27.599954 13772 ts_tablet_manager.cc:1397] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 07:00:27.600653 13775 log.cc:824] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:27.601121 13773 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: No bootstrap required, opened a new log
I20260620 07:00:27.601449 13773 ts_tablet_manager.cc:1397] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260620 07:00:27.602447 13609 heartbeater.cc:503] Master 127.0.97.62:39663 was elected leader, sending a full tablet report...
I20260620 07:00:27.610035 13772 raft_consensus.cc:348] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.610682 13772 raft_consensus.cc:374] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:27.610925 13772 raft_consensus.cc:729] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71dd9102b2e14c0db0102294ef8d3025, State: Initialized, Role: FOLLOWER
I20260620 07:00:27.611645 13772 consensus_queue.cc:260] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.612131 13775 tablet_bootstrap.cc:492] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: No bootstrap required, opened a new log
I20260620 07:00:27.612473 13775 ts_tablet_manager.cc:1397] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260620 07:00:27.612859 13744 heartbeater.cc:503] Master 127.0.97.62:39663 was elected leader, sending a full tablet report...
I20260620 07:00:27.612905 13773 raft_consensus.cc:348] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.613324 13773 raft_consensus.cc:374] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:27.613415 13773 raft_consensus.cc:729] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd32429e6b3e4e0ab6be3d1f55b7ee19, State: Initialized, Role: FOLLOWER
I20260620 07:00:27.613582 13772 ts_tablet_manager.cc:1428] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260620 07:00:27.614082 13773 consensus_queue.cc:260] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.615765 13773 ts_tablet_manager.cc:1428] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Time spent starting tablet: real 0.014s user 0.010s sys 0.001s
I20260620 07:00:27.621218 13775 raft_consensus.cc:348] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.621628 13775 raft_consensus.cc:374] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:27.621722 13775 raft_consensus.cc:729] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6003d59783495a9bcbacf2c7b27294, State: Initialized, Role: FOLLOWER
I20260620 07:00:27.622211 13775 consensus_queue.cc:260] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.623180 13465 heartbeater.cc:503] Master 127.0.97.62:39663 was elected leader, sending a full tablet report...
I20260620 07:00:27.623567 13775 ts_tablet_manager.cc:1428] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s
W20260620 07:00:27.683405 13745 tablet.cc:2369] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:27.696830 13466 tablet.cc:2369] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:27.811137 13610 tablet.cc:2369] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:27.882323 13780 raft_consensus.cc:482] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:00:27.882684 13780 raft_consensus.cc:504] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.884524 13780 leader_election.cc:288] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71dd9102b2e14c0db0102294ef8d3025 (127.0.97.3:40699), bd32429e6b3e4e0ab6be3d1f55b7ee19 (127.0.97.2:45685)
I20260620 07:00:27.891317 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.885060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51643 (local address 127.0.97.3:40699)
0620 07:00:27.885279 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.885286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.885992 (+ 706us) server_negotiation.cc:407] Connection header received
0620 07:00:27.886214 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.886218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.886274 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.886356 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:27.887012 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.887700 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.889608 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.890368 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.890480 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.890607 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.890949 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.891079 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:00:27.891323 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.884998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35701 (local address 127.0.97.2:45685)
0620 07:00:27.885218 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.885225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.885983 (+ 758us) server_negotiation.cc:407] Connection header received
0620 07:00:27.886163 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.886168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.886250 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.886318 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:27.886972 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.887683 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.889605 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.890379 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.890502 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.890625 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.890962 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.891098 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:00:27.891464 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.884991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45685 (local address 127.0.97.1:35701)
0620 07:00:27.885843 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.885856 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.886016 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.886369 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.886373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.886398 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:27.886784 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.886797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.887814 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.887819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.889450 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.889462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.890669 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.890674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.890806 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.890827 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.890918 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.891046 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":58,"thread_start_us":501,"threads_started":1}
I20260620 07:00:27.891520 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.884804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40699 (local address 127.0.97.1:51643)
0620 07:00:27.885803 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:27.885819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:27.886075 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:27.886401 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:27.886405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:27.886430 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:27.886854 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.886872 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.887842 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.887845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.889456 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:27.889469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.890629 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.890635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:27.890767 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.890815 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:00:27.890899 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:00:27.891029 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":59,"spinlock_wait_cycles":26496,"thread_start_us":621,"threads_started":1}
I20260620 07:00:27.892442 13699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7738aebe84b70a44e6281e87a2789" candidate_uuid: "9d6003d59783495a9bcbacf2c7b27294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71dd9102b2e14c0db0102294ef8d3025" is_pre_election: true
I20260620 07:00:27.892441 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7738aebe84b70a44e6281e87a2789" candidate_uuid: "9d6003d59783495a9bcbacf2c7b27294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" is_pre_election: true
I20260620 07:00:27.892872 13699 raft_consensus.cc:2454] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6003d59783495a9bcbacf2c7b27294 in term 0.
I20260620 07:00:27.892876 13564 raft_consensus.cc:2454] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d6003d59783495a9bcbacf2c7b27294 in term 0.
I20260620 07:00:27.893458 13355 leader_election.cc:302] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 71dd9102b2e14c0db0102294ef8d3025, 9d6003d59783495a9bcbacf2c7b27294, bd32429e6b3e4e0ab6be3d1f55b7ee19; no voters:
I20260620 07:00:27.893831 13780 raft_consensus.cc:2781] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:00:27.893962 13780 raft_consensus.cc:482] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:00:27.894057 13780 raft_consensus.cc:3037] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:27.896174 13780 raft_consensus.cc:504] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.896790 13780 leader_election.cc:288] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [CANDIDATE]: Term 1 election: Requested vote from peers 71dd9102b2e14c0db0102294ef8d3025 (127.0.97.3:40699), bd32429e6b3e4e0ab6be3d1f55b7ee19 (127.0.97.2:45685)
I20260620 07:00:27.897162 13699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7738aebe84b70a44e6281e87a2789" candidate_uuid: "9d6003d59783495a9bcbacf2c7b27294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71dd9102b2e14c0db0102294ef8d3025"
I20260620 07:00:27.897284 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3f7738aebe84b70a44e6281e87a2789" candidate_uuid: "9d6003d59783495a9bcbacf2c7b27294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19"
I20260620 07:00:27.897328 13699 raft_consensus.cc:3037] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:27.897473 13564 raft_consensus.cc:3037] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:27.899541 13699 raft_consensus.cc:2454] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6003d59783495a9bcbacf2c7b27294 in term 1.
I20260620 07:00:27.899672 13564 raft_consensus.cc:2454] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d6003d59783495a9bcbacf2c7b27294 in term 1.
I20260620 07:00:27.899904 13355 leader_election.cc:302] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71dd9102b2e14c0db0102294ef8d3025, 9d6003d59783495a9bcbacf2c7b27294; no voters:
I20260620 07:00:27.900147 13780 raft_consensus.cc:2781] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:27.900586 13780 raft_consensus.cc:686] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6003d59783495a9bcbacf2c7b27294, State: Running, Role: LEADER
I20260620 07:00:27.901077 13780 consensus_queue.cc:237] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } }
I20260620 07:00:27.905984 13267 catalog_manager.cc:5495] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d6003d59783495a9bcbacf2c7b27294 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9d6003d59783495a9bcbacf2c7b27294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6003d59783495a9bcbacf2c7b27294" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 } health_report { overall_health: UNKNOWN } } }
I20260620 07:00:27.949386 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:27.943827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.0.97.1:42741)
0620 07:00:27.944052 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:27.944057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:27.944077 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:00:27.944268 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:27.944272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:27.944375 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:27.944501 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:00:27.944853 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.945571 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.947130 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:27.947537 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:27.947639 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:27.948213 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:00:27.948742 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:00:27.948849 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:27.949017 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 07:00:27.949088 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 07:00:27.958271 13699 raft_consensus.cc:1261] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 1 FOLLOWER]: Refusing update from remote peer 9d6003d59783495a9bcbacf2c7b27294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:27.958329 13564 raft_consensus.cc:1261] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 1 FOLLOWER]: Refusing update from remote peer 9d6003d59783495a9bcbacf2c7b27294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:27.959270 13783 consensus_queue.cc:1035] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71dd9102b2e14c0db0102294ef8d3025" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:27.959599 13780 consensus_queue.cc:1035] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd32429e6b3e4e0ab6be3d1f55b7ee19" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:27.970659 13786 mvcc.cc:205] Tried to move back new op lower bound from 7298821439305666560 to 7298821439086612480. Current Snapshot: MvccSnapshot[applied={T|T < 7298821439305666560}]
I20260620 07:00:27.985780 13268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:27.986074 13268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:27.989799 13268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 819df99890714e14ac7be6489267625a: Sending DeleteTablet for 3 replicas of tablet a3f7738aebe84b70a44e6281e87a2789
I20260620 07:00:27.990953 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7738aebe84b70a44e6281e87a2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60756
I20260620 07:00:27.990978 13679 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7738aebe84b70a44e6281e87a2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54202
I20260620 07:00:27.991201 13544 tablet_service.cc:1514] Processing DeleteTablet for tablet a3f7738aebe84b70a44e6281e87a2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38412
I20260620 07:00:27.998787 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13305
I20260620 07:00:28.003480 13795 tablet_replica.cc:330] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: stopping tablet replica
I20260620 07:00:28.004138 13797 tablet_replica.cc:330] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: stopping tablet replica
I20260620 07:00:28.004136 13795 raft_consensus.cc:2229] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:28.004638 13795 raft_consensus.cc:2258] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:28.004808 13796 tablet_replica.cc:330] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: stopping tablet replica
I20260620 07:00:28.005337 13797 raft_consensus.cc:2229] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:28.005825 13796 raft_consensus.cc:2229] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:00:28.006369 13797 raft_consensus.cc:2258] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:28.006897 13795 ts_tablet_manager.cc:1905] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:28.007340 13796 raft_consensus.cc:2258] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:28.009482 13797 ts_tablet_manager.cc:1905] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:28.011426 13796 ts_tablet_manager.cc:1905] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:28.013629 13795 ts_tablet_manager.cc:1918] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:28.013849 13795 log.cc:1196] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/a3f7738aebe84b70a44e6281e87a2789
I20260620 07:00:28.014400 13795 ts_tablet_manager.cc:1939] T a3f7738aebe84b70a44e6281e87a2789 P 71dd9102b2e14c0db0102294ef8d3025: Deleting consensus metadata
I20260620 07:00:28.016600 13254 catalog_manager.cc:4841] TS 71dd9102b2e14c0db0102294ef8d3025 (127.0.97.3:40699): tablet a3f7738aebe84b70a44e6281e87a2789 (table test-table [id=4921937fd2414751bf44655fc28c5f78]) successfully deleted
I20260620 07:00:28.018069 13797 ts_tablet_manager.cc:1918] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:28.018303 13797 log.cc:1196] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/a3f7738aebe84b70a44e6281e87a2789
I20260620 07:00:28.018893 13797 ts_tablet_manager.cc:1939] T a3f7738aebe84b70a44e6281e87a2789 P bd32429e6b3e4e0ab6be3d1f55b7ee19: Deleting consensus metadata
I20260620 07:00:28.020718 13252 catalog_manager.cc:4841] TS bd32429e6b3e4e0ab6be3d1f55b7ee19 (127.0.97.2:45685): tablet a3f7738aebe84b70a44e6281e87a2789 (table test-table [id=4921937fd2414751bf44655fc28c5f78]) successfully deleted
I20260620 07:00:28.021997 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.000022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.97.1:42741)
0620 07:00:28.000189 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:28.000195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:28.000414 (+ 219us) server_negotiation.cc:407] Connection header received
0620 07:00:28.000494 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:28.000499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:28.000580 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:28.000680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:28.006565 (+ 5885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.007784 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.009722 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.010501 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.010644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:28.014034 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:28.014048 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:28.014069 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:28.014118 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:28.018262 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.019292 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.019302 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.019307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.019500 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.019968 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.019973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.019977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.020441 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:28.020729 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:28.021370 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 07:00:28.021631 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:00:28.022873 13460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:28.121644 13796 ts_tablet_manager.cc:1918] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:28.121948 13796 log.cc:1196] T a3f7738aebe84b70a44e6281e87a2789 P 9d6003d59783495a9bcbacf2c7b27294: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/a3f7738aebe84b70a44e6281e87a2789
I20260620 07:00:28.122246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13305
W20260620 07:00:28.136196 13255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42741 (error 108)
I20260620 07:00:28.136948 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13469
I20260620 07:00:28.138387 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.137411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60786)
0620 07:00:28.138035 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.138171 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":207,"threads_started":1}
W20260620 07:00:28.138669 13255 catalog_manager.cc:4584] TS 9d6003d59783495a9bcbacf2c7b27294 (127.0.97.1:42741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3f7738aebe84b70a44e6281e87a2789: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
I20260620 07:00:28.150059 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.137656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.0.97.2:45685)
0620 07:00:28.137840 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:28.137846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:28.137940 (+ 94us) server_negotiation.cc:407] Connection header received
0620 07:00:28.138110 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:28.138114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:28.138179 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:28.138265 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:28.139628 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.140341 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.141641 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.142041 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.142131 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:28.144488 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:28.144503 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:28.144519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:28.144552 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:28.147275 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.148064 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.148072 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.148074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.148224 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.148605 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.148610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.148612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.149026 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:28.149263 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:28.149580 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 07:00:28.149750 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:00:28.150791 13604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:28.192732 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.192178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60792)
0620 07:00:28.192423 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.192552 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 07:00:28.193922 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.193472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60802)
0620 07:00:28.193655 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.193745 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 07:00:28.239648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13469
I20260620 07:00:28.253175 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13613
I20260620 07:00:28.254313 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.253822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60810)
0620 07:00:28.254035 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.254134 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 07:00:28.255249 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.254904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60814)
0620 07:00:28.255056 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.255126 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 07:00:28.266244 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.253967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.0.97.3:40699)
0620 07:00:28.254122 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:28.254127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:28.254225 (+ 98us) server_negotiation.cc:407] Connection header received
0620 07:00:28.254399 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:28.254404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:28.254510 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:28.254630 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:28.256034 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.256743 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.258016 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:28.258410 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:28.258500 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:28.260955 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:28.260970 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:28.260986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:28.261019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:28.263763 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.264449 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.264457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.264459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.264582 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:28.264873 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:28.264876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:28.264878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:28.265272 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:28.265496 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:28.265777 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 07:00:28.265938 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:00:28.267124 13739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:28.335919 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.335448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60820)
0620 07:00:28.335656 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.335741 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 07:00:28.336987 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:28.336556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42741 (local address 127.0.0.1:60828)
0620 07:00:28.336763 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:28.336836 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 07:00:28.356400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13613
I20260620 07:00:28.369632 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13235
I20260620 07:00:28.370820 13298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:28.481383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13235
I20260620 07:00:28.498831 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 07:00:28.834522 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 07:00:28.834789 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 07:00:29.170586 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 07:00:29.170 UTC [12287] LOG: received smart shutdown request
2026-06-20 07:00:29.173 UTC [12287] LOG: background worker "logical replication launcher" (PID 12296) exited with exit code 1
2026-06-20 07:00:29.173 UTC [12291] LOG: shutting down
2026-06-20 07:00:29.349 UTC [12287] LOG: database system is shut down
2026-06-20T07:00:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50967 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:00:29 dist-test-slave-mz20 krb5kdc[13813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:00:29 dist-test-slave-mz20 krb5kdc[13813](info): set up 2 sockets
Jun 20 07:00:29 dist-test-slave-mz20 krb5kdc[13813](info): commencing operation
krb5kdc: starting...
W20260620 07:00:30.478834 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:00:30 dist-test-slave-mz20 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:00:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:00:30.702710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:31.039921 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:31.040243 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:31.040377 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:31.048282 13826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:31.048416 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:31.048470 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:31.048552 13826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:31.048609 13826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:31.058862 13826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:31.060214 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:31.061535 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:31.072849 13832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:31.072861 13831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:31.072911 13834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:31.073807 13826 server_base.cc:1034] running on GCE node
I20260620 07:00:31.074647 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:31.076063 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:31.077327 13826 hybrid_clock.cc:648] HybridClock initialized: now 1781938831077233 us; error 34 us; skew 500 ppm
I20260620 07:00:31.077714 13826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:31.084697 13826 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:31.086283 13826 webserver.cc:460] Webserver started at http://127.0.97.62:35709/ using document root <none> and password file <none>
I20260620 07:00:31.087064 13826 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:31.087183 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:31.087476 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:31.090508 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "8b399fb772714cae86e7f1a88e0f6606"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "7dca644fdb1de1f68c3f87fb937d04a3"
server_key_iv: "6fa7117088b9e0804a2dd6c64843d0ff"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.091349 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "8b399fb772714cae86e7f1a88e0f6606"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "7dca644fdb1de1f68c3f87fb937d04a3"
server_key_iv: "6fa7117088b9e0804a2dd6c64843d0ff"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.096900 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:00:31.100500 13841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:31.101862 13826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:00:31.102036 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "8b399fb772714cae86e7f1a88e0f6606"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "7dca644fdb1de1f68c3f87fb937d04a3"
server_key_iv: "6fa7117088b9e0804a2dd6c64843d0ff"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.102223 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:31.121843 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:31.126195 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:31.126466 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:31.144428 13826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33451
I20260620 07:00:31.144457 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33451 every 8 connection(s)
I20260620 07:00:31.145884 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:31.149637 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:31.154594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13826
I20260620 07:00:31.154830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:00:31.155470 388 external_mini_cluster.cc:1428] Setting key 57e04e65f137cbdca615add1b9572e89
I20260620 07:00:31.156600 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 07:00:31.179642 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.159676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.0.97.62:33451)
0620 07:00:31.161337 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:31.161345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:31.161398 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:00:31.162888 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:31.162894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:31.163124 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:31.163316 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:31.164536 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.165946 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.167359 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.167899 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.168071 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.172102 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:31.172110 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:31.172123 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:31.172158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:31.175078 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.175866 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.175873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.175875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.176061 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.176442 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.176445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.176447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.176967 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:31.177133 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:31.177709 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 07:00:31.178097 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":1446,"spinlock_wait_cycles":33664,"thread_start_us":1244,"threads_started":1}
I20260620 07:00:31.179675 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.159675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33451 (local address 127.0.0.1:35296)
0620 07:00:31.160508 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.160562 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:31.162496 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:31.163335 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:31.163342 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:31.164024 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:31.164353 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.164371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.166084 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.166088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.167229 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.167242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.168117 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.168123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.168368 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.169015 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:31.169040 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:31.171876 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:31.175220 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.175231 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.175245 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.175727 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.176167 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.176172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.176176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.176328 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.177092 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:31.177100 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:31.177326 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 07:00:31.177551 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 07:00:31.178221 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":209,"thread_start_us":361,"threads_started":1}
I20260620 07:00:31.182885 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:00:31.184087 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.183631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60692)
0620 07:00:31.183799 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.183929 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:00:31.184438 13843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
I20260620 07:00:31.186028 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.185635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60700)
0620 07:00:31.185785 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.185867 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 07:00:31.186246 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 07:00:31.202226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45565
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:31.235635 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } attempt: 1
I20260620 07:00:31.236944 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.236393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60704)
0620 07:00:31.236611 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.236722 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:00:31.238090 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.237675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60718)
0620 07:00:31.237827 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.237924 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:00:31.238314 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
I20260620 07:00:31.306655 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } attempt: 2
I20260620 07:00:31.307972 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.307393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60728)
0620 07:00:31.307667 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.307778 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:00:31.309099 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.308672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60742)
0620 07:00:31.308820 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.308937 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:00:31.309329 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
I20260620 07:00:31.400581 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } attempt: 3
I20260620 07:00:31.401796 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.401261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60752)
0620 07:00:31.401520 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.401623 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 07:00:31.402894 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.402438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60766)
0620 07:00:31.402618 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.402717 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:00:31.403147 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
W20260620 07:00:31.525403 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:31.525708 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:31.525799 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:31.533179 13900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:31.533300 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:31.533346 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:31.533401 13900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:31.533449 13900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:31.543665 13900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:31.544970 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:31.546236 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:00:31.546470 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } attempt: 4
I20260620 07:00:31.547783 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.547178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60782)
0620 07:00:31.547519 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.547623 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 07:00:31.548952 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.548517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60786)
0620 07:00:31.548678 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.548789 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:00:31.549209 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45565: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45565: connect: Connection refused (error 111)
W20260620 07:00:31.559033 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:31.559044 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:31.559044 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:31.559744 13900 server_base.cc:1034] running on GCE node
I20260620 07:00:31.560561 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:31.561993 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:31.563184 13900 hybrid_clock.cc:648] HybridClock initialized: now 1781938831563123 us; error 52 us; skew 500 ppm
I20260620 07:00:31.563540 13900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:31.566953 13900 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 07:00:31.568446 13900 webserver.cc:460] Webserver started at http://127.0.97.61:42523/ using document root <none> and password file <none>
I20260620 07:00:31.569144 13900 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:31.569244 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:31.569500 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:31.572638 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data/instance:
uuid: "b51b2f9d4eed433e86f61abb042eb571"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "db967b64be50043616c3566871fa8f44"
server_key_iv: "711a2a6d480ee3340fab859b3d32ce87"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.573423 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal/instance:
uuid: "b51b2f9d4eed433e86f61abb042eb571"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "db967b64be50043616c3566871fa8f44"
server_key_iv: "711a2a6d480ee3340fab859b3d32ce87"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.578977 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 07:00:31.582664 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:31.584134 13900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:00:31.584311 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "b51b2f9d4eed433e86f61abb042eb571"
format_stamp: "Formatted at 2026-06-20 07:00:31 on dist-test-slave-mz20"
server_key: "db967b64be50043616c3566871fa8f44"
server_key_iv: "711a2a6d480ee3340fab859b3d32ce87"
server_key_version: "encryptionkey@0"
I20260620 07:00:31.584511 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:31.606144 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:31.610806 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:31.611129 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:31.629133 13900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45565
I20260620 07:00:31.629172 13966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45565 every 8 connection(s)
I20260620 07:00:31.630436 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
I20260620 07:00:31.633249 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13900
I20260620 07:00:31.633419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-1/wal/instance
I20260620 07:00:31.633827 388 external_mini_cluster.cc:1428] Setting key f1bc514e947a2e1c3ce97c425bd0a56e
I20260620 07:00:31.634234 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:31.640743 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:31.661671 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.643487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.97.62:33451)
0620 07:00:31.643657 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:31.643661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:31.645189 (+ 1528us) server_negotiation.cc:407] Connection header received
0620 07:00:31.647250 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:31.647256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:31.647353 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:31.647451 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:31.649066 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.649742 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.651399 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.651811 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.651927 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.655712 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:31.655718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:31.655722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:31.655760 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:31.658484 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.659327 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.659332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.659335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.659463 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.659909 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.659913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.659915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.660204 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:31.660359 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:31.661098 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 07:00:31.661347 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:00:31.662739 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.644128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33451 (local address 127.0.0.1:35304)
0620 07:00:31.644994 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.645042 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:31.646935 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:31.647730 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:31.647739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:31.648295 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:31.648885 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.648904 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.649910 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.649914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.651245 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.651259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.652042 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.652050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.652277 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.652941 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:31.652965 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:31.655432 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:31.658674 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.658684 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.658705 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.659180 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.659578 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.659582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.659586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.659782 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.660332 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:31.660341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:31.660714 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 07:00:31.660975 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 07:00:31.661497 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":361,"threads_started":1}
I20260620 07:00:31.664631 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 07:00:31.680224 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41857
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:31.681918 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.665389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60794)
0620 07:00:31.665599 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.665620 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:31.665749 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:31.666570 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:31.666575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:31.666936 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:31.667097 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.667105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.669010 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.669014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.669840 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.669851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.671030 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.671034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.671201 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.671729 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:31.671745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:31.675496 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:31.679331 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.679336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.679338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.679787 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.680179 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.680189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.680193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.680336 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.680983 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:31.680987 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:31.681134 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 07:00:31.681250 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 07:00:31.681533 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:00:31.681998 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.665433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.97.61:45565)
0620 07:00:31.666059 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:31.666066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:31.666096 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:00:31.666148 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:31.666156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:31.666384 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:31.666522 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:31.667254 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.668855 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.669988 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.670887 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.671023 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.675803 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:31.675821 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:31.675841 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:31.675878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:31.679181 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.679920 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.679930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.679933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.680075 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.680459 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.680464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.680477 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.680835 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:31.681047 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:31.681406 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 07:00:31.681638 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":299,"threads_started":1}
I20260620 07:00:31.684717 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:00:31.685940 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.685436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34938)
0620 07:00:31.685615 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.685754 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:00:31.686213 13918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.687809 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.687438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34952)
0620 07:00:31.687598 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.687682 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:00:31.688009 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.737309 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 1
I20260620 07:00:31.738495 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.738044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34954)
0620 07:00:31.738228 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.738332 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 07:00:31.739400 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.739087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34958)
0620 07:00:31.739220 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.739298 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:00:31.739624 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.807875 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 2
I20260620 07:00:31.809038 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.808564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34968)
0620 07:00:31.808767 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.808852 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 07:00:31.810050 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.809729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34980)
0620 07:00:31.809846 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.809934 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 07:00:31.810250 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.848526 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } attempt: 5
Jun 20 07:00:31 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:00:31.864076 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.849215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60796)
0620 07:00:31.849552 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.849568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:31.849676 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:31.850123 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:31.850127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:31.850526 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:31.850759 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.850770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.852186 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.852191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.853123 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:31.853136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.854153 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.854158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:31.854312 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.855019 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:31.855036 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:31.857283 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:31.861415 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.861420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.861422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.861851 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.862303 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:31.862307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:31.862309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:31.862501 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:31.863232 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:31.863237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:31.863403 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 07:00:31.863567 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 07:00:31.863807 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260620 07:00:31.864270 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.849403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.61:45565)
0620 07:00:31.849634 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:31.849640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:31.849668 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:00:31.849785 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:31.849791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:31.849912 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:31.850050 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:31.850934 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.852000 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.853279 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:31.853964 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:31.854114 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:31.857518 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:31.857525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:31.857531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:31.857578 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:31.861197 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.861998 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.862006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.862010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.862168 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:31.862647 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:31.862652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:31.862655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:31.863062 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:31.863254 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:31.863710 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 07:00:31.863956 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:00:31.865208 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:00:31.866370 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.865979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34984)
0620 07:00:31.866138 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.866225 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 07:00:31.867345 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.867032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:34996)
0620 07:00:31.867166 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.867248 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:00:31.867532 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.901458 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 3
I20260620 07:00:31.902631 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.902141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35010)
0620 07:00:31.902354 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.902443 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:00:31.903620 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.903219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35020)
0620 07:00:31.903429 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.903515 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 07:00:31.903816 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.918769 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 1
I20260620 07:00:31.919827 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.919374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35030)
0620 07:00:31.919569 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.919654 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 07:00:31.920737 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.920420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35044)
0620 07:00:31.920562 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.920629 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 07:00:31.920940 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:31.989127 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 2
I20260620 07:00:31.990269 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.989796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35054)
0620 07:00:31.990012 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.990100 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 07:00:31.991240 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:31.990900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35058)
0620 07:00:31.991063 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:31.991130 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:00:31.991439 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
W20260620 07:00:32.004982 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:32.005330 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:32.005463 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:32.012858 13973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:32.012985 13973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:32.013065 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:32.013144 13973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:32.013203 13973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:32.023016 13973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33451,127.0.97.61:45565,127.0.97.60:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:32.024323 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:32.025626 13973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:32.037082 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:32.037060 13982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:32.037238 13979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:32.037736 13973 server_base.cc:1034] running on GCE node
I20260620 07:00:32.038597 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:32.040040 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:32.041276 13973 hybrid_clock.cc:648] HybridClock initialized: now 1781938832041223 us; error 44 us; skew 500 ppm
I20260620 07:00:32.041672 13973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:32.044788 13973 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 07:00:32.046303 13973 webserver.cc:460] Webserver started at http://127.0.97.60:34621/ using document root <none> and password file <none>
I20260620 07:00:32.047091 13973 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:32.047006 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 4
I20260620 07:00:32.047221 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:32.047524 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:32.048146 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.047633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35062)
0620 07:00:32.047867 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.047966 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:00:32.049166 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.048805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35066)
0620 07:00:32.048950 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.049033 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:00:32.049366 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:32.051132 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data/instance:
uuid: "96c57f81aba6431bb65e5044dea82cf4"
format_stamp: "Formatted at 2026-06-20 07:00:32 on dist-test-slave-mz20"
server_key: "e2a2c511f89165589ff2af411a447d31"
server_key_iv: "cdf591c17e8344c48bc53a043c6e570f"
server_key_version: "encryptionkey@0"
I20260620 07:00:32.051872 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal/instance:
uuid: "96c57f81aba6431bb65e5044dea82cf4"
format_stamp: "Formatted at 2026-06-20 07:00:32 on dist-test-slave-mz20"
server_key: "e2a2c511f89165589ff2af411a447d31"
server_key_iv: "cdf591c17e8344c48bc53a043c6e570f"
server_key_version: "encryptionkey@0"
I20260620 07:00:32.057009 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:00:32.060554 13989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:32.062039 13973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:00:32.062202 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "96c57f81aba6431bb65e5044dea82cf4"
format_stamp: "Formatted at 2026-06-20 07:00:32 on dist-test-slave-mz20"
server_key: "e2a2c511f89165589ff2af411a447d31"
server_key_iv: "cdf591c17e8344c48bc53a043c6e570f"
server_key_version: "encryptionkey@0"
I20260620 07:00:32.062393 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:32.078159 13973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:32.082481 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:32.082768 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:32.097627 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 3
I20260620 07:00:32.098834 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.098271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35070)
0620 07:00:32.098500 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.098635 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:00:32.099975 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.099520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35074)
0620 07:00:32.099729 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.099824 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 07:00:32.100234 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41857: connect: Connection refused (error 111)
I20260620 07:00:32.100808 13973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41857
I20260620 07:00:32.100837 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41857 every 8 connection(s)
I20260620 07:00:32.102164 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
I20260620 07:00:32.105762 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:32.111864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 13973
I20260620 07:00:32.112005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781938634772379-388-0/minicluster-data/master-2/wal/instance
I20260620 07:00:32.112463 388 external_mini_cluster.cc:1428] Setting key c888ef3bd2bb4f72b5d8856b306e571b
I20260620 07:00:32.112326 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:00:32.131366 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.115081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.0.97.62:33451)
0620 07:00:32.115297 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.115304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.115981 (+ 677us) server_negotiation.cc:407] Connection header received
0620 07:00:32.117360 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.117364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.117430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.117525 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.118843 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.119566 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.121253 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.121888 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.122031 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.125568 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.125573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.125577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.125610 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.128370 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.129095 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.129101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.129104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.129230 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.129638 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.129643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.129646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.129945 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.130109 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.130851 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.131094 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:00:32.132565 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.115056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33451 (local address 127.0.0.1:35310)
0620 07:00:32.115824 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.115869 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:32.117091 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:32.117601 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:32.117607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:32.118286 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:32.118677 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.118696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.119711 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.119717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.121121 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.121132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.122147 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.122155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.122433 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.123213 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:32.123235 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:32.125353 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:32.128501 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.128516 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.128529 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.128978 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.129333 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.129338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.129342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.129526 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.130096 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:32.130102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:32.130410 (+ 308us) client_negotiation.cc:769] Sending connection context
0620 07:00:32.130756 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 07:00:32.131181 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":315,"threads_started":1}
I20260620 07:00:32.132807 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.114288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.97.60:41857)
0620 07:00:32.115060 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.115075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.115125 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:00:32.117081 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.117109 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.117406 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.117584 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.118995 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.120392 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.122030 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.122899 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.122991 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.127021 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.127030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.127043 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.127073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.129756 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.130507 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.130515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.130585 (+ 70us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.130726 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.131110 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.131115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.131119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.131504 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.131789 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.132152 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.132391 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":351,"threads_started":1}
I20260620 07:00:32.134193 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:32.147761 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.134993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45565 (local address 127.0.0.1:60810)
0620 07:00:32.135153 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.135170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:32.135255 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:32.135551 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:32.135555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:32.135977 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:32.136159 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.136168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.137151 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.137156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.138138 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.138150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.138867 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.138874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.139060 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.139651 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:32.139668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:32.141759 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:32.145054 (+ 3295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.145059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.145063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.145566 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.145992 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.145997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.146001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.146168 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.146800 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:32.146806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:32.146965 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 07:00:32.147112 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:00:32.147402 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:00:32.147789 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.134903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.97.61:45565)
0620 07:00:32.135070 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.135076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.135230 (+ 154us) server_negotiation.cc:407] Connection header received
0620 07:00:32.135341 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.135347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.135418 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.135508 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.136309 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.137000 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.138278 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.138732 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.138825 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.141969 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.141974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.141977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.142012 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.144821 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.145694 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.145701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.145705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.145855 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.146286 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.146292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.146295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.146688 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.146844 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.147229 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.147434 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:00:32.149044 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:00:32.150738 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.135830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.97.62:33451)
0620 07:00:32.136009 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.136013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.136037 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:00:32.136078 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.136082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.136134 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.136204 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.137419 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.138143 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.139543 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.139969 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.140078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.144050 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.144057 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.144062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.144108 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.147366 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.148420 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.148425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.148429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.148621 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.148956 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.148961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.148963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.149311 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.149597 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.150172 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.150373 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:00:32.162384 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.149864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35092)
0620 07:00:32.150122 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.150139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:32.150234 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:32.150666 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:32.150670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:32.151232 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:32.151409 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.151421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.152528 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.152532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.153565 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.153576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.154220 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.154225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.154396 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.154966 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:32.154982 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:32.157084 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:32.160146 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.160151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.160155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.160531 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.160882 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.160888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.160891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.161017 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.161517 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:32.161521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:32.161667 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 07:00:32.161817 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 07:00:32.162117 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 07:00:32.162659 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.149807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.97.60:41857)
0620 07:00:32.149967 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.149972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.150312 (+ 340us) server_negotiation.cc:407] Connection header received
0620 07:00:32.150346 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.150350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.150402 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.150478 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.151547 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.152395 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.153684 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.154094 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.154195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.157269 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.157275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.157279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.157317 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.160017 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.160635 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.160641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.160644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.160753 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.161114 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.161120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.161123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.161409 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.161536 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.161931 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.162276 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":48}
I20260620 07:00:32.166532 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.152586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.97.61:45565)
0620 07:00:32.152742 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.152748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.152765 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:00:32.152812 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.152818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.152886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.152978 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.154337 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.155117 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.156671 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.157138 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.157267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.161247 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.161252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.161256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.161289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.164098 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.164725 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.164731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.164734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.164862 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.165223 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.165227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.165229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.165481 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.165741 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.166017 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.166238 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:00:32.167883 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: Bootstrap starting.
I20260620 07:00:32.170877 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:00:32.171173 14041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:32.172144 14041 log.cc:824] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:32.174938 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: No bootstrap required, opened a new log
I20260620 07:00:32.186196 14041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.186702 14041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:32.186815 14041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c57f81aba6431bb65e5044dea82cf4, State: Initialized, Role: FOLLOWER
I20260620 07:00:32.187527 14041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.189565 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.175451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.97.61:45565)
0620 07:00:32.175628 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.175634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.176140 (+ 506us) server_negotiation.cc:407] Connection header received
0620 07:00:32.176367 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.176371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.176431 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.176572 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.178116 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.178863 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.180400 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.180927 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.181039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.183884 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.183889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.183893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.183926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.186391 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.187312 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.187319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.187322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.187511 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.187922 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.187927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.187930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.188262 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.188426 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.188907 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.189226 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:00:32.194643 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.195129 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.197005 14041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:32.205472 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.175830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.0.97.62:33451)
0620 07:00:32.176000 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.176005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.189699 (+ 13694us) server_negotiation.cc:407] Connection header received
0620 07:00:32.189948 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.189956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.190045 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.190120 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.192422 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.193187 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.194593 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.195230 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.195329 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.198532 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.198561 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.198566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.198610 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.202003 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.202818 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.202824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.202827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.202989 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.203381 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.203386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.203389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.203755 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.204007 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.204861 (+ 854us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.205112 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:00:32.207393 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.181929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35104 (local address 127.0.97.60:41857)
0620 07:00:32.182158 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.182164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.190915 (+ 8751us) server_negotiation.cc:407] Connection header received
0620 07:00:32.191964 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.191971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.192060 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.192132 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.194292 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.195524 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.197501 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.198265 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.199170 (+ 905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.201207 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.201213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.201217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.201259 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.204484 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.205264 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.205269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.205277 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.205459 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.205906 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.205910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.205913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.206248 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.206499 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.206808 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.207025 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:00:32.204523 13973 master_runner.cc:186] Error getting master registration for 127.0.97.62:33451: OK, instance_id { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" instance_seqno: 1781938831139014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:00:32.216969 13973 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:00:32.217165 14065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:00:32.217283 14065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:00:32.277484 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 4
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:00:32.293552 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.278158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35110)
0620 07:00:32.278380 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.278400 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:32.278580 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:32.278972 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:32.278976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:32.279352 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:32.279566 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.279576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.281345 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.281350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.282882 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.282898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.283634 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.283640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.283831 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.284439 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:32.284483 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:32.287143 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:32.291085 (+ 3942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.291089 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.291092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.291469 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.291906 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.291909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.291912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.292074 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.292681 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:32.292685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:32.292832 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 07:00:32.292999 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 07:00:32.293263 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:00:32.293736 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.278250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35110 (local address 127.0.97.60:41857)
0620 07:00:32.278444 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.278463 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.278487 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:00:32.278709 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.278714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.278815 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.278931 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.279712 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.280956 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.283046 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.283494 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.283597 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.287374 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.287380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.287384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.287448 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.290933 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.291597 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.291606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.291610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.291783 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.292187 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.292192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.292196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.292566 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.292709 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.293118 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.293357 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:00:32.299623 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: Bootstrap starting.
I20260620 07:00:32.303761 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:32.305130 13893 log.cc:824] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:32.308554 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: No bootstrap required, opened a new log
I20260620 07:00:32.319497 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.319937 13893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:32.320055 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b399fb772714cae86e7f1a88e0f6606, State: Initialized, Role: FOLLOWER
I20260620 07:00:32.320745 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.321683 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.322023 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.323069 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:00:32.334860 13995 tablet.cc:2369] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:32.335745 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:00:32.335942 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:00:32.333959 13826 master_runner.cc:186] Error getting master registration for 127.0.97.61:45565: OK, instance_id { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" instance_seqno: 1781938831623983 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:00:32.347103 13826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:00:32.348588 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } attempt: 5
Jun 20 07:00:32 dist-test-slave-mz20 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:00:32.370003 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.349218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41857 (local address 127.0.0.1:35112)
0620 07:00:32.349519 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:32.349540 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:32.349677 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:32.350223 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:32.350229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:32.350963 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:32.351179 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.351193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.352740 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.352746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.354220 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:32.354237 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.356255 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.356262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:32.356464 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.357160 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:32.357185 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:32.362803 (+ 5618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:32.367128 (+ 4325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.367133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.367137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.367698 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.368145 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:32.368149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:32.368152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:32.368323 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:32.368962 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:32.368968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:32.369142 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 07:00:32.369298 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 07:00:32.369599 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260620 07:00:32.370060 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:32.349347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.97.60:41857)
0620 07:00:32.349549 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:32.349556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:32.349605 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:00:32.349840 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:32.349846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:32.349924 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:32.350016 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:32.351376 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.352539 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.355186 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:32.355898 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:32.356033 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:32.363044 (+ 7011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:32.363052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:32.363057 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:32.363098 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:32.366988 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.367851 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.367857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.367861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.368020 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:32.368457 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:32.368462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:32.368465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:32.368829 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:32.368981 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:32.369428 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 07:00:32.369684 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:00:32.376591 13967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: Bootstrap starting.
W20260620 07:00:32.379423 13847 tablet.cc:2369] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:32.381062 13967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:32.382431 13967 log.cc:824] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:32.388530 13967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: No bootstrap required, opened a new log
I20260620 07:00:32.399623 13967 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.400113 13967 raft_consensus.cc:374] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:32.400238 13967 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51b2f9d4eed433e86f61abb042eb571, State: Initialized, Role: FOLLOWER
I20260620 07:00:32.400959 13967 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.402000 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.402489 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.403373 13967 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:32.424522 13900 master_runner.cc:263] No leader master found from master b51b2f9d4eed433e86f61abb042eb571
I20260620 07:00:32.424682 13900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 07:00:32.427273 14093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:00:32.427556 14093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:00:32.492242 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:00:32.492589 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.494591 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b399fb772714cae86e7f1a88e0f6606 (127.0.97.62:33451), b51b2f9d4eed433e86f61abb042eb571 (127.0.97.61:45565)
I20260620 07:00:32.495805 13942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96c57f81aba6431bb65e5044dea82cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51b2f9d4eed433e86f61abb042eb571" is_pre_election: true
I20260620 07:00:32.496526 13942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c57f81aba6431bb65e5044dea82cf4 in term 0.
I20260620 07:00:32.498129 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96c57f81aba6431bb65e5044dea82cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b399fb772714cae86e7f1a88e0f6606" is_pre_election: true
I20260620 07:00:32.499140 13991 leader_election.cc:302] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96c57f81aba6431bb65e5044dea82cf4, b51b2f9d4eed433e86f61abb042eb571; no voters:
I20260620 07:00:32.499919 13868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96c57f81aba6431bb65e5044dea82cf4 in term 0.
I20260620 07:00:32.501082 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:00:32.501237 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:00:32.501335 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:32.505323 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.511482 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96c57f81aba6431bb65e5044dea82cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b399fb772714cae86e7f1a88e0f6606"
I20260620 07:00:32.511751 13868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:32.515600 13868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c57f81aba6431bb65e5044dea82cf4 in term 1.
I20260620 07:00:32.516225 13990 leader_election.cc:302] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b399fb772714cae86e7f1a88e0f6606, 96c57f81aba6431bb65e5044dea82cf4; no voters:
I20260620 07:00:32.520380 13942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96c57f81aba6431bb65e5044dea82cf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51b2f9d4eed433e86f61abb042eb571"
I20260620 07:00:32.520643 13942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:32.524406 13942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96c57f81aba6431bb65e5044dea82cf4 in term 1.
I20260620 07:00:32.525352 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [CANDIDATE]: Term 1 election: Requested vote from peers 8b399fb772714cae86e7f1a88e0f6606 (127.0.97.62:33451), b51b2f9d4eed433e86f61abb042eb571 (127.0.97.61:45565)
I20260620 07:00:32.525794 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:32.526280 14054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [term 1 LEADER]: Becoming Leader. State: Replica: 96c57f81aba6431bb65e5044dea82cf4, State: Running, Role: LEADER
I20260620 07:00:32.527786 14054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } }
I20260620 07:00:32.545629 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c57f81aba6431bb65e5044dea82cf4. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.546198 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:32.553395 14097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:32.556003 14097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:32.575987 13942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [term 1 FOLLOWER]: Refusing update from remote peer 96c57f81aba6431bb65e5044dea82cf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:32.577096 14095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:32.584769 13868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [term 1 FOLLOWER]: Refusing update from remote peer 96c57f81aba6431bb65e5044dea82cf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:32.585755 14095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:32.612288 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c57f81aba6431bb65e5044dea82cf4. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.612674 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 07:00:32.613933 13921 tablet.cc:2369] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:32.616128 14098 mvcc.cc:205] Tried to move back new op lower bound from 7298821458216136704 to 7298821458046005248. Current Snapshot: MvccSnapshot[applied={T|T < 7298821458216136704}]
I20260620 07:00:32.617069 14099 mvcc.cc:205] Tried to move back new op lower bound from 7298821458216136704 to 7298821458046005248. Current Snapshot: MvccSnapshot[applied={T|T < 7298821458216136704}]
I20260620 07:00:32.617524 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c57f81aba6431bb65e5044dea82cf4. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.617871 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.623378 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.623672 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:32.623924 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.624114 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:32.624588 14097 catalog_manager.cc:1347] Generated new cluster ID: ec5f330af4fa4974aa4ae96641656951
I20260620 07:00:32.624676 14097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:32.627363 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.627604 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.642673 14097 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:00:32.644388 14097 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:32.650276 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96c57f81aba6431bb65e5044dea82cf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b399fb772714cae86e7f1a88e0f6606" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33451 } } peers { permanent_uuid: "b51b2f9d4eed433e86f61abb042eb571" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45565 } } peers { permanent_uuid: "96c57f81aba6431bb65e5044dea82cf4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41857 } } }
I20260620 07:00:32.650599 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:00:32.684389 14097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96c57f81aba6431bb65e5044dea82cf4: Generated new TSK 0
I20260620 07:00:32.685590 14097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:00:33.338670 14079 catalog_manager.cc:1259] Loaded cluster ID: ec5f330af4fa4974aa4ae96641656951
I20260620 07:00:33.338804 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: loading cluster ID for follower catalog manager: success
I20260620 07:00:33.342288 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: acquiring CA information for follower catalog manager: success
I20260620 07:00:33.344636 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b399fb772714cae86e7f1a88e0f6606: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:00:33.430218 14093 catalog_manager.cc:1259] Loaded cluster ID: ec5f330af4fa4974aa4ae96641656951
I20260620 07:00:33.430356 14093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: loading cluster ID for follower catalog manager: success
I20260620 07:00:33.433303 14093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: acquiring CA information for follower catalog manager: success
I20260620 07:00:33.435568 14093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b51b2f9d4eed433e86f61abb042eb571: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:00:33.769984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13826
I20260620 07:00:33.771647 13888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:33.896823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13826
I20260620 07:00:33.918171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13900
I20260620 07:00:33.919505 13962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:34.038079 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13900
I20260620 07:00:34.057688 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 13973
I20260620 07:00:34.058933 14036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:34.179683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 13973
2026-06-20T07:00:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4727 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:00:34 dist-test-slave-mz20 krb5kdc[14115](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:00:34 dist-test-slave-mz20 krb5kdc[14115](info): set up 2 sockets
Jun 20 07:00:34 dist-test-slave-mz20 krb5kdc[14115](info): commencing operation
krb5kdc: starting...
W20260620 07:00:35.222841 388 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:00:35 dist-test-slave-mz20 krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:00:35.446310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:35.772027 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:35.772308 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:35.772397 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:35.779536 14128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:35.779646 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:35.779698 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:35.779754 14128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:35.779796 14128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:35.789667 14128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:35.790951 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:35.792212 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:35.803460 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:35.803498 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:35.803517 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:35.804420 14128 server_base.cc:1034] running on GCE node
I20260620 07:00:35.805182 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:35.806654 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:35.807880 14128 hybrid_clock.cc:648] HybridClock initialized: now 1781938835807828 us; error 42 us; skew 500 ppm
I20260620 07:00:35.808221 14128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:35 dist-test-slave-mz20 krb5kdc[14115](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:35.811295 14128 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:35.812733 14128 webserver.cc:460] Webserver started at http://127.0.97.62:33685/ using document root <none> and password file <none>
I20260620 07:00:35.813423 14128 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:35.813535 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:35.813830 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:35.816798 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "97537405d4554002b4c3ca3a4a84c79a"
format_stamp: "Formatted at 2026-06-20 07:00:35 on dist-test-slave-mz20"
server_key: "4fb24d8bb5ae8065bfb45ea8457f2962"
server_key_iv: "c83bb858cb775051342462087cb2d5a0"
server_key_version: "encryptionkey@0"
I20260620 07:00:35.817515 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "97537405d4554002b4c3ca3a4a84c79a"
format_stamp: "Formatted at 2026-06-20 07:00:35 on dist-test-slave-mz20"
server_key: "4fb24d8bb5ae8065bfb45ea8457f2962"
server_key_iv: "c83bb858cb775051342462087cb2d5a0"
server_key_version: "encryptionkey@0"
I20260620 07:00:35.822496 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:00:35.826081 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:35.827380 14128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:35.827548 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "97537405d4554002b4c3ca3a4a84c79a"
format_stamp: "Formatted at 2026-06-20 07:00:35 on dist-test-slave-mz20"
server_key: "4fb24d8bb5ae8065bfb45ea8457f2962"
server_key_iv: "c83bb858cb775051342462087cb2d5a0"
server_key_version: "encryptionkey@0"
I20260620 07:00:35.827714 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:35.859658 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:35.863998 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:35.864282 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:35.881650 14128 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39585
I20260620 07:00:35.881680 14194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39585 every 8 connection(s)
I20260620 07:00:35.883064 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:35.887015 14195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:35.887665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14128
I20260620 07:00:35.887871 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:00:35.888545 388 external_mini_cluster.cc:1428] Setting key 659867a19f84aa4f959e74826f550348
I20260620 07:00:35.901571 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a: Bootstrap starting.
Jun 20 07:00:35 dist-test-slave-mz20 krb5kdc[14115](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:35.904819 14195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:35.906139 14195 log.cc:824] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:35.909576 14195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a: No bootstrap required, opened a new log
I20260620 07:00:35.913648 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:35.891057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.0.97.62:39585)
0620 07:00:35.891896 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:35.891912 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:35.891969 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:00:35.893915 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:35.893949 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:35.894526 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:35.895029 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:35.896393 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:35.898332 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:35.899753 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:35.900772 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:35.900952 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:35.905056 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:35.905068 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:35.905087 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:35.905143 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:35.908896 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:35.909670 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:35.909678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:35.909686 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:35.909882 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:35.910308 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:35.910314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:35.910317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:35.911040 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:35.911352 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:35.911746 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 07:00:35.912210 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":343,"threads_started":1}
I20260620 07:00:35.917677 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:00:35.919127 14195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260620 07:00:35.919495 14195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:35.919600 14195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97537405d4554002b4c3ca3a4a84c79a, State: Initialized, Role: FOLLOWER
I20260620 07:00:35.920238 14195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260620 07:00:35.920507 14195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:35.920655 14195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:35.920806 14195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:35.923074 14195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260620 07:00:35.923568 14195 leader_election.cc:302] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97537405d4554002b4c3ca3a4a84c79a; no voters:
I20260620 07:00:35.924120 14195 leader_election.cc:288] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:00:35.924240 14200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:35.924858 14200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [term 1 LEADER]: Becoming Leader. State: Replica: 97537405d4554002b4c3ca3a4a84c79a, State: Running, Role: LEADER
I20260620 07:00:35.925410 14200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } }
I20260620 07:00:35.926038 14195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:35.928154 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97537405d4554002b4c3ca3a4a84c79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } } }
I20260620 07:00:35.928143 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97537405d4554002b4c3ca3a4a84c79a. Latest consensus state: current_term: 1 leader_uuid: "97537405d4554002b4c3ca3a4a84c79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97537405d4554002b4c3ca3a4a84c79a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39585 } } }
I20260620 07:00:35.928634 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:35.928732 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:35.930482 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:35.937374 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:35.946352 14208 catalog_manager.cc:1347] Generated new cluster ID: 7aec290f5bb04dafbb5fe6dfd9587a9e
I20260620 07:00:35.946480 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:35.957244 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:00:35.958798 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:35.968997 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97537405d4554002b4c3ca3a4a84c79a: Generated new TSK 0
I20260620 07:00:35.969938 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:00:35.978410 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14128
I20260620 07:00:35.979715 14190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:36.069367 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14128
2026-06-20T07:00:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1882 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:00:36 dist-test-slave-mz20 krb5kdc[14223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:00:36 dist-test-slave-mz20 krb5kdc[14223](info): set up 2 sockets
Jun 20 07:00:36 dist-test-slave-mz20 krb5kdc[14223](info): commencing operation
krb5kdc: starting...
W20260620 07:00:37.078805 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:00:37 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:00:37.293948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:37.617184 14236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:37.617506 14236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:37.617651 14236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:37.625106 14236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:37.625237 14236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:37.625316 14236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:37.625388 14236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:37.625447 14236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:37.635898 14236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:37.637305 14236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:37.638734 14236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:37.650498 14241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:37.650486 14242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:37.650527 14244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:37.651473 14236 server_base.cc:1034] running on GCE node
I20260620 07:00:37.652355 14236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:37.653954 14236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:37.655231 14236 hybrid_clock.cc:648] HybridClock initialized: now 1781938837655116 us; error 59 us; skew 500 ppm
I20260620 07:00:37.655604 14236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:37 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:37.658864 14236 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:37.660532 14236 webserver.cc:460] Webserver started at http://127.0.97.62:42231/ using document root <none> and password file <none>
I20260620 07:00:37.661231 14236 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:37.661348 14236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:37.661643 14236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:37.665190 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "20ba7a9e8fe2464588420cdb5078754e"
format_stamp: "Formatted at 2026-06-20 07:00:37 on dist-test-slave-mz20"
server_key: "99b2627f8e267fb58c0da3bd1be68c24"
server_key_iv: "78dc9cd93a2d3ddf0ea4c91aa36b6093"
server_key_version: "encryptionkey@0"
I20260620 07:00:37.665985 14236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "20ba7a9e8fe2464588420cdb5078754e"
format_stamp: "Formatted at 2026-06-20 07:00:37 on dist-test-slave-mz20"
server_key: "99b2627f8e267fb58c0da3bd1be68c24"
server_key_iv: "78dc9cd93a2d3ddf0ea4c91aa36b6093"
server_key_version: "encryptionkey@0"
I20260620 07:00:37.671211 14236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:00:37.674685 14251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:37.676183 14236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:00:37.676380 14236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "20ba7a9e8fe2464588420cdb5078754e"
format_stamp: "Formatted at 2026-06-20 07:00:37 on dist-test-slave-mz20"
server_key: "99b2627f8e267fb58c0da3bd1be68c24"
server_key_iv: "78dc9cd93a2d3ddf0ea4c91aa36b6093"
server_key_version: "encryptionkey@0"
I20260620 07:00:37.676565 14236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:37.690173 14236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:37.694332 14236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:37.694651 14236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:37.712031 14236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37161
I20260620 07:00:37.712051 14302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37161 every 8 connection(s)
I20260620 07:00:37.713451 14236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:37.715377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14236
I20260620 07:00:37.715595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:00:37.716413 388 external_mini_cluster.cc:1428] Setting key b3984855a40c559fa627899731cca60e
I20260620 07:00:37.718084 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:00:37 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:37.731705 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap starting.
I20260620 07:00:37.734983 14303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:37.736229 14303 log.cc:824] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:37.739511 14303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: No bootstrap required, opened a new log
I20260620 07:00:37.739858 14306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:37.718471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.0.97.62:37161)
0620 07:00:37.719258 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:37.719270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:37.719309 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:00:37.720878 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:37.720901 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:37.721336 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:37.721694 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:37.723457 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:37.724948 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:37.726737 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:37.727490 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:37.727640 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:37.732472 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:37.732481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:37.732496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:37.732534 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:37.735568 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:37.736403 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:37.736412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:37.736420 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:37.736591 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:37.737007 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:37.737012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:37.737015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:37.737733 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:37.737953 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:37.738410 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 07:00:37.738832 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":381,"threads_started":1}
I20260620 07:00:37.746819 14303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:37.747184 14303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:37.747277 14303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Initialized, Role: FOLLOWER
I20260620 07:00:37.747815 14303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:37.748025 14303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:37.748148 14303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:37.748297 14303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:37.750417 14303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:37.750988 14303 leader_election.cc:302] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ba7a9e8fe2464588420cdb5078754e; no voters:
I20260620 07:00:37.751551 14303 leader_election.cc:288] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:00:37.751688 14308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:37.752333 14308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 LEADER]: Becoming Leader. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Running, Role: LEADER
I20260620 07:00:37.752878 14308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:37.753611 14303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:37.755626 14309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:37.755998 14309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:37.756479 14310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ba7a9e8fe2464588420cdb5078754e. Latest consensus state: current_term: 1 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:37.756786 14310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:37.757809 14314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:37.762362 14314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:37.771145 14314 catalog_manager.cc:1347] Generated new cluster ID: 5522f388de90430b81e2b02247bde665
I20260620 07:00:37.771274 14314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:37.792423 14314 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:00:37.793757 14314 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:37.802572 14314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Generated new TSK 0
I20260620 07:00:37.803788 14314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:00:37.866065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:38.198786 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:38.199066 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:38.199158 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:38.206377 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:38.206491 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:38.206640 14329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:00:38.216517 14329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:38.217729 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:38.219050 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:38.231519 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:38.231611 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:38.231755 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:38.232590 14329 server_base.cc:1034] running on GCE node
I20260620 07:00:38.233227 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:38.234181 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:38.235376 14329 hybrid_clock.cc:648] HybridClock initialized: now 1781938838235324 us; error 40 us; skew 500 ppm
I20260620 07:00:38.235677 14329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:38 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:38.238965 14329 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:00:38.240540 14329 webserver.cc:460] Webserver started at http://127.0.97.1:36911/ using document root <none> and password file <none>
I20260620 07:00:38.241223 14329 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:38.241343 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:38.241633 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:38.244602 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "d91e9fbfadc44ef2870569b65cd7736f"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "084ce8c88d85567ffc60abd17df63ad7"
server_key_iv: "86ee374d8261475fbba4a6f3a194559d"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.245369 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d91e9fbfadc44ef2870569b65cd7736f"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "084ce8c88d85567ffc60abd17df63ad7"
server_key_iv: "86ee374d8261475fbba4a6f3a194559d"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.250702 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:00:38.254237 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.255600 14329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:38.255771 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d91e9fbfadc44ef2870569b65cd7736f"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "084ce8c88d85567ffc60abd17df63ad7"
server_key_iv: "86ee374d8261475fbba4a6f3a194559d"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.255952 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:38.291270 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:38.295588 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:38.295866 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:38.296829 14329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:38.298178 14329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:38.298256 14329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.298384 14329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:38.298436 14329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.328842 14329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39275
I20260620 07:00:38.328888 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39275 every 8 connection(s)
I20260620 07:00:38.330132 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:00:38.330525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14329
I20260620 07:00:38.330766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:00:38.331729 388 external_mini_cluster.cc:1428] Setting key 2266c2e2a7af7c55d64a81fb57dc10fd
Jun 20 07:00:38 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:38.350870 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:38.333036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36717 (local address 127.0.97.62:37161)
0620 07:00:38.333529 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:38.333535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:38.334535 (+ 1000us) server_negotiation.cc:407] Connection header received
0620 07:00:38.336457 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:38.336462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:38.336547 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:38.336658 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:38.338318 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.339286 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.340949 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.341540 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.341656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:38.345454 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:38.345459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:38.345462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:38.345495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:38.347902 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:38.348623 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:38.348630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:38.348632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:38.348750 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:38.349151 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:38.349156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:38.349160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:38.349427 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:38.349556 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:38.350254 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 07:00:38.350484 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
I20260620 07:00:38.351846 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:38.333546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:36717)
0620 07:00:38.334360 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:38.334412 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:38.336157 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:38.336955 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:38.336965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:38.337520 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:38.338130 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:38.338151 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.339468 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.339472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:38.340779 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:38.340793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.341788 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.341794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:38.342074 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:38.342798 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:38.342832 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:38.345231 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:38.348048 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:38.348058 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:38.348074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:38.348506 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:38.348853 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:38.348858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:38.348862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:38.349046 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:38.349544 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:38.349552 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:38.349897 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 07:00:38.350139 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 07:00:38.350711 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":341,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 07:00:38.353430 14457 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:38.353749 14457 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:38.354554 14457 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 07:00:38.356791 14268 ts_manager.cc:194] Registered new tserver with Master: d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275)
I20260620 07:00:38.358237 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:00:38.381738 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:38.697139 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:38.697435 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:38.697553 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:38.704977 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:38.705086 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:38.705242 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:00:38.717311 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:38.718916 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:38.720525 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:38.734647 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:38.734951 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:38.735013 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:38.735529 14463 server_base.cc:1034] running on GCE node
I20260620 07:00:38.736145 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:38.737125 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:38.738330 14463 hybrid_clock.cc:648] HybridClock initialized: now 1781938838738280 us; error 45 us; skew 500 ppm
I20260620 07:00:38.738677 14463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:38 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:38.741678 14463 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:00:38.743062 14463 webserver.cc:460] Webserver started at http://127.0.97.2:43779/ using document root <none> and password file <none>
I20260620 07:00:38.743692 14463 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:38.743785 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:38.744036 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:38.746848 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "7af97e8a5a67424b9cb6916a9df7b8cf"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "637e8121f458b28af3352580a79b9427"
server_key_iv: "e0cc3c9c2fda6b38370bf263e242bd9c"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.747560 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7af97e8a5a67424b9cb6916a9df7b8cf"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "637e8121f458b28af3352580a79b9427"
server_key_iv: "e0cc3c9c2fda6b38370bf263e242bd9c"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.752537 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:00:38.756193 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.757606 14463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:00:38.757771 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "7af97e8a5a67424b9cb6916a9df7b8cf"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "637e8121f458b28af3352580a79b9427"
server_key_iv: "e0cc3c9c2fda6b38370bf263e242bd9c"
server_key_version: "encryptionkey@0"
I20260620 07:00:38.757961 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:38.787107 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:38.791375 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:38.791639 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:38.792728 14463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:38.794087 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:38.794163 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.794287 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:38.794338 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:38.823500 14463 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40469
I20260620 07:00:38.823546 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40469 every 8 connection(s)
I20260620 07:00:38.824705 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:00:38.833015 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14463
I20260620 07:00:38.833187 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:00:38.833797 388 external_mini_cluster.cc:1428] Setting key 4954ab0bde7298a0d91f0faa8db1be0d
Jun 20 07:00:38 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:38.845137 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:38.827330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49351 (local address 127.0.97.62:37161)
0620 07:00:38.827508 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:38.827513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:38.828604 (+ 1091us) server_negotiation.cc:407] Connection header received
0620 07:00:38.830341 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:38.830346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:38.830415 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:38.830510 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:38.831910 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.832638 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.834528 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.835107 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.835216 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:38.839076 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:38.839083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:38.839088 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:38.839129 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:38.842065 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:38.842844 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:38.842850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:38.842854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:38.843026 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:38.843520 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:38.843523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:38.843525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:38.843780 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:38.843883 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:38.844707 (+ 824us) server_negotiation.cc:299] Negotiation successful
0620 07:00:38.844882 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:00:38.846462 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:38.827750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:49351)
0620 07:00:38.828442 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:38.828487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:38.830060 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:38.830730 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:38.830738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:38.831217 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:38.831748 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:38.831764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.832780 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.832785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:38.834367 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:38.834383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:38.835356 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:38.835365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:38.835663 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:38.836468 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:38.836491 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:38.838826 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:38.842226 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:38.842233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:38.842246 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:38.842738 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:38.843129 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:38.843134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:38.843138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:38.843427 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:38.843892 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:38.843900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:38.844335 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 07:00:38.844613 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 07:00:38.845133 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":275,"threads_started":1}
I20260620 07:00:38.847904 14591 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:38.848194 14591 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:38.848879 14591 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
I20260620 07:00:38.850237 14268 ts_manager.cc:194] Registered new tserver with Master: 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2:40469)
I20260620 07:00:38.851046 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49351
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:00:38.881729 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:39.192379 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:39.192688 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:39.192807 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:39.200412 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:39.200580 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:39.200739 14596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:00:39.210443 14596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:39.211761 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:39.213017 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:39.225265 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:39.225327 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:39.225304 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:39.226043 14596 server_base.cc:1034] running on GCE node
I20260620 07:00:39.226652 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:39.227550 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:39.228768 14596 hybrid_clock.cc:648] HybridClock initialized: now 1781938839228718 us; error 37 us; skew 500 ppm
I20260620 07:00:39.229095 14596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:39 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:39.232164 14596 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:00:39.233529 14596 webserver.cc:460] Webserver started at http://127.0.97.3:43275/ using document root <none> and password file <none>
I20260620 07:00:39.234187 14596 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:39.234297 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:39.234603 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:00:39.237459 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
format_stamp: "Formatted at 2026-06-20 07:00:39 on dist-test-slave-mz20"
server_key: "b3168cf00da08004c7a898502809fac6"
server_key_iv: "691186771a08c1bd8ae4a7bf933daeb7"
server_key_version: "encryptionkey@0"
I20260620 07:00:39.238224 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
format_stamp: "Formatted at 2026-06-20 07:00:39 on dist-test-slave-mz20"
server_key: "b3168cf00da08004c7a898502809fac6"
server_key_iv: "691186771a08c1bd8ae4a7bf933daeb7"
server_key_version: "encryptionkey@0"
I20260620 07:00:39.243327 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:00:39.246668 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:39.248052 14596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:39.248219 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
format_stamp: "Formatted at 2026-06-20 07:00:39 on dist-test-slave-mz20"
server_key: "b3168cf00da08004c7a898502809fac6"
server_key_iv: "691186771a08c1bd8ae4a7bf933daeb7"
server_key_version: "encryptionkey@0"
I20260620 07:00:39.248401 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:39.263460 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:39.267819 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:39.268085 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:39.269062 14596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:39.270347 14596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:39.270416 14596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:39.270512 14596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:39.270601 14596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 07:00:39.297889 14596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37079
I20260620 07:00:39.297933 14723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37079 every 8 connection(s)
I20260620 07:00:39.299165 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:00:39.302457 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14596
I20260620 07:00:39.302655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:00:39.303201 388 external_mini_cluster.cc:1428] Setting key 993ca6da278aaa2eed82b27a0223d0ec
Jun 20 07:00:39 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:39.318727 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.301585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58691 (local address 127.0.97.62:37161)
0620 07:00:39.301755 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.301760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.302932 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 07:00:39.305024 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.305029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.305108 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.305186 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:39.306890 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.307668 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.309224 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.309592 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.309678 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.312983 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:39.312988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:39.312992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:39.313034 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:39.315526 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.316241 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.316245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.316248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.316403 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.316847 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.316851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.316854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.317178 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:39.317313 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.318177 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.318386 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:00:39.319638 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.301917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:58691)
0620 07:00:39.302756 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.302807 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.304753 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.305484 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.305497 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.306101 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:39.306726 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.306746 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.307851 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.307855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.309094 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.309107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.309767 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.309772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.309997 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.310628 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:39.310652 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:39.312775 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:39.315670 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:39.315677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:39.315690 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:39.316133 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:39.316530 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:39.316535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:39.316537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:39.316732 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:39.317319 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:39.317327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:39.317721 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.318031 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.318566 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":277,"threads_started":1}
I20260620 07:00:39.320940 14724 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:39.321201 14724 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:39.321877 14724 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
I20260620 07:00:39.323226 14268 ts_manager.cc:194] Registered new tserver with Master: cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079)
I20260620 07:00:39.323985 14268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:58691
I20260620 07:00:39.330744 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:00:39.348474 14459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.335544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.0.97.62:37161)
0620 07:00:39.335768 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.335774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.336041 (+ 267us) server_negotiation.cc:407] Connection header received
0620 07:00:39.336218 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.336223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.336293 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.336376 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:39.337711 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.338458 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.339848 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.340285 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.340398 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.342774 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:39.342780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:39.342783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:39.342818 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:39.345621 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.346382 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.346387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.346389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.346522 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.346967 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.346974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.346976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.347327 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:39.347576 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.347919 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.348155 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 07:00:39.352752 14268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:00:39.359885 14268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:00:39.361835 14457 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:39.384402 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.376067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37079 (local address 127.0.0.1:47832)
0620 07:00:39.376596 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.376610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.376694 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.378136 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.378144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.378174 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:39.378499 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.378508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.379862 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.379867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.381835 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.381849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.383327 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.383334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.383516 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.383576 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.383672 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.383894 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":3,"thread_start_us":275,"threads_started":1}
I20260620 07:00:39.384989 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.376424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.97.3:37079)
0620 07:00:39.377104 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.377110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.377136 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:00:39.377214 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.377219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.377640 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.377834 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:39.378678 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.379751 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.381978 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.383131 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.384189 (+ 1058us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.384430 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.384508 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.384667 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":425,"threads_started":1}
I20260620 07:00:39.388449 14659 tablet_service.cc:1467] Processing CreateTablet for tablet 74852c49f5fc43b4a1a0388f124c607b (DEFAULT_TABLE table=test-table [id=863760e5c8a545538f83891c87717144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:39.390106 14659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74852c49f5fc43b4a1a0388f124c607b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:39.390627 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.372396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37034)
0620 07:00:39.373192 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.373239 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.373440 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.383894 (+ 10454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.383899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.383922 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:39.384410 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.384422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.386563 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.386568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.388471 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.388480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.389807 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.389813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.389989 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.390033 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.390109 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.390244 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":1867,"threads_started":1}
I20260620 07:00:39.390676 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.373073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.0.97.1:39275)
0620 07:00:39.383381 (+ 10308us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.383387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.383410 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:00:39.383484 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.383489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.383712 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.383916 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:39.384778 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.386374 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.388633 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.389596 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.389711 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.389928 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.390178 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.390326 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10154,"thread_start_us":10011,"threads_started":1}
I20260620 07:00:39.390754 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.372544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40469 (local address 127.0.0.1:34400)
0620 07:00:39.374648 (+ 2104us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.374666 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.374781 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.383632 (+ 8851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.383637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.383661 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:39.384060 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.384073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.385992 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.385998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.388090 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.388105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.389918 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.389925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.390103 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.390136 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.390204 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.390326 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1927,"mutex_wait_us":1,"thread_start_us":642,"threads_started":1}
I20260620 07:00:39.391803 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.372988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.97.2:40469)
0620 07:00:39.382994 (+ 10006us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.383004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.383034 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:00:39.383129 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.383137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.383410 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.383619 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:39.384330 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.385793 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.388270 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.389686 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.391089 (+ 1403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.391270 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.391333 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.391500 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9841,"thread_start_us":9713,"threads_started":1}
I20260620 07:00:39.394865 14526 tablet_service.cc:1467] Processing CreateTablet for tablet 74852c49f5fc43b4a1a0388f124c607b (DEFAULT_TABLE table=test-table [id=863760e5c8a545538f83891c87717144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:39.395008 14392 tablet_service.cc:1467] Processing CreateTablet for tablet 74852c49f5fc43b4a1a0388f124c607b (DEFAULT_TABLE table=test-table [id=863760e5c8a545538f83891c87717144]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:39.396028 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74852c49f5fc43b4a1a0388f124c607b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:39.396420 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74852c49f5fc43b4a1a0388f124c607b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:39.405900 14742 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Bootstrap starting.
I20260620 07:00:39.409740 14744 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Bootstrap starting.
I20260620 07:00:39.409847 14742 tablet_bootstrap.cc:654] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:39.411082 14743 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Bootstrap starting.
I20260620 07:00:39.411619 14742 log.cc:824] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:39.413343 14744 tablet_bootstrap.cc:654] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:39.414556 14743 tablet_bootstrap.cc:654] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:39.414628 14744 log.cc:824] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:39.415057 14742 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: No bootstrap required, opened a new log
I20260620 07:00:39.415303 14742 ts_tablet_manager.cc:1397] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 07:00:39.415825 14743 log.cc:824] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:39.418527 14743 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: No bootstrap required, opened a new log
I20260620 07:00:39.418818 14743 ts_tablet_manager.cc:1397] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260620 07:00:39.418998 14744 tablet_bootstrap.cc:492] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: No bootstrap required, opened a new log
I20260620 07:00:39.419274 14744 ts_tablet_manager.cc:1397] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 07:00:39.426018 14742 raft_consensus.cc:348] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.426560 14742 raft_consensus.cc:374] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:39.426703 14742 raft_consensus.cc:729] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd265dde62314bdb8ebadcbcaa593dc0, State: Initialized, Role: FOLLOWER
I20260620 07:00:39.426753 14743 raft_consensus.cc:348] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.427255 14743 raft_consensus.cc:374] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:39.427356 14743 raft_consensus.cc:729] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af97e8a5a67424b9cb6916a9df7b8cf, State: Initialized, Role: FOLLOWER
I20260620 07:00:39.427387 14742 consensus_queue.cc:260] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.428046 14743 consensus_queue.cc:260] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.429152 14724 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:39.429404 14744 raft_consensus.cc:348] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.429857 14742 ts_tablet_manager.cc:1428] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260620 07:00:39.429894 14744 raft_consensus.cc:374] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:39.430014 14744 raft_consensus.cc:729] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91e9fbfadc44ef2870569b65cd7736f, State: Initialized, Role: FOLLOWER
I20260620 07:00:39.430696 14744 consensus_queue.cc:260] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.430955 14591 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:39.431658 14743 ts_tablet_manager.cc:1428] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260620 07:00:39.432569 14744 ts_tablet_manager.cc:1428] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260620 07:00:39.550736 14725 tablet.cc:2369] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:39.576715 14592 tablet.cc:2369] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:39.582803 14458 tablet.cc:2369] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:39.630649 14749 raft_consensus.cc:482] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:00:39.630940 14749 raft_consensus.cc:504] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.632592 14749 leader_election.cc:288] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275), cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079)
I20260620 07:00:39.641505 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.633092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55537 (local address 127.0.97.1:39275)
0620 07:00:39.633288 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.633294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.634135 (+ 841us) server_negotiation.cc:407] Connection header received
0620 07:00:39.634287 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.634292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.634360 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.634441 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:39.635236 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.636382 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.639148 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.640377 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.640524 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.640634 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.641096 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.641238 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:00:39.641537 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.633123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56985 (local address 127.0.97.3:37079)
0620 07:00:39.633291 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.633296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.634032 (+ 736us) server_negotiation.cc:407] Connection header received
0620 07:00:39.634256 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.634261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.634343 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.634449 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:39.635236 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.636382 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.639155 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.640376 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.640523 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.640634 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.641096 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.641238 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:00:39.641728 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.632966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.97.2:55537)
0620 07:00:39.634041 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.634062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.634159 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.634463 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.634467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.634493 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:39.635009 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.635024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.636508 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.636513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.638933 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.638953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.640692 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.640697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.640881 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.640952 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.641045 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.641253 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":172,"spinlock_wait_cycles":18176,"thread_start_us":567,"threads_started":1}
I20260620 07:00:39.641752 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.632873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37079 (local address 127.0.97.2:56985)
0620 07:00:39.633905 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.633923 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:39.634096 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:39.634463 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:39.634467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:39.634591 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:39.635051 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.635060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.636511 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.636515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.638958 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:39.638968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.640681 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.640689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:39.640877 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.640948 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 07:00:39.641045 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:00:39.641254 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":145,"spinlock_wait_cycles":25344,"thread_start_us":635,"threads_started":1}
I20260620 07:00:39.643062 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74852c49f5fc43b4a1a0388f124c607b" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" is_pre_election: true
I20260620 07:00:39.643062 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74852c49f5fc43b4a1a0388f124c607b" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91e9fbfadc44ef2870569b65cd7736f" is_pre_election: true
I20260620 07:00:39.643643 14679 raft_consensus.cc:2454] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 0.
I20260620 07:00:39.643644 14412 raft_consensus.cc:2454] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 0.
I20260620 07:00:39.644413 14480 leader_election.cc:302] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7af97e8a5a67424b9cb6916a9df7b8cf, d91e9fbfadc44ef2870569b65cd7736f; no voters:
I20260620 07:00:39.644825 14749 raft_consensus.cc:2781] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:00:39.644959 14749 raft_consensus.cc:482] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:00:39.645045 14749 raft_consensus.cc:3037] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:39.647239 14749 raft_consensus.cc:504] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.647810 14749 leader_election.cc:288] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 election: Requested vote from peers d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275), cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079)
I20260620 07:00:39.648381 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74852c49f5fc43b4a1a0388f124c607b" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
I20260620 07:00:39.648381 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74852c49f5fc43b4a1a0388f124c607b" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91e9fbfadc44ef2870569b65cd7736f"
I20260620 07:00:39.648597 14679 raft_consensus.cc:3037] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:39.648622 14412 raft_consensus.cc:3037] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:39.651741 14679 raft_consensus.cc:2454] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 1.
I20260620 07:00:39.651736 14412 raft_consensus.cc:2454] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 1.
I20260620 07:00:39.652181 14480 leader_election.cc:302] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7af97e8a5a67424b9cb6916a9df7b8cf, d91e9fbfadc44ef2870569b65cd7736f; no voters:
I20260620 07:00:39.652527 14749 raft_consensus.cc:2781] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:39.652982 14749 raft_consensus.cc:686] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 LEADER]: Becoming Leader. State: Replica: 7af97e8a5a67424b9cb6916a9df7b8cf, State: Running, Role: LEADER
I20260620 07:00:39.653528 14749 consensus_queue.cc:237] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } }
I20260620 07:00:39.657929 14267 catalog_manager.cc:5495] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } health_report { overall_health: UNKNOWN } } }
I20260620 07:00:39.680256 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.675098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.0.97.2:40469)
0620 07:00:39.675290 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.675294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.675312 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:00:39.675486 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.675490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.675582 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.675707 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:00:39.676102 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.676784 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.678126 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.678522 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.678652 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.679098 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:00:39.679642 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:00:39.679749 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.679899 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.679971 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:00:39.688699 14679 raft_consensus.cc:1261] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Refusing update from remote peer 7af97e8a5a67424b9cb6916a9df7b8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:39.688699 14412 raft_consensus.cc:1261] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Refusing update from remote peer 7af97e8a5a67424b9cb6916a9df7b8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:39.689599 14753 consensus_queue.cc:1035] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:39.689990 14749 consensus_queue.cc:1035] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:39.705968 14758 mvcc.cc:205] Tried to move back new op lower bound from 7298821487353954304 to 7298821487224238080. Current Snapshot: MvccSnapshot[applied={T|T < 7298821487353954304}]
I20260620 07:00:39.731647 14267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:39.731920 14267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:39.735595 14267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Sending DeleteTablet for 3 replicas of tablet 74852c49f5fc43b4a1a0388f124c607b
I20260620 07:00:39.736785 14392 tablet_service.cc:1514] Processing DeleteTablet for tablet 74852c49f5fc43b4a1a0388f124c607b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37034
I20260620 07:00:39.736792 14526 tablet_service.cc:1514] Processing DeleteTablet for tablet 74852c49f5fc43b4a1a0388f124c607b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34400
I20260620 07:00:39.736953 14659 tablet_service.cc:1514] Processing DeleteTablet for tablet 74852c49f5fc43b4a1a0388f124c607b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47832
I20260620 07:00:39.738082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14329
I20260620 07:00:39.749655 14767 tablet_replica.cc:330] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: stopping tablet replica
I20260620 07:00:39.750264 14767 raft_consensus.cc:2229] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:39.750859 14767 raft_consensus.cc:2258] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:39.755272 14767 ts_tablet_manager.cc:1905] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:39.756105 14768 tablet_replica.cc:330] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: stopping tablet replica
I20260620 07:00:39.756644 14768 raft_consensus.cc:2229] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:39.757253 14768 raft_consensus.cc:2258] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:39.758844 14766 tablet_replica.cc:330] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: stopping tablet replica
I20260620 07:00:39.759485 14768 ts_tablet_manager.cc:1905] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:39.759486 14766 raft_consensus.cc:2229] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:00:39.760738 14766 raft_consensus.cc:2258] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 07:00:39 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:00:39.763216 14766 ts_tablet_manager.cc:1905] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:39.766443 14768 ts_tablet_manager.cc:1918] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:39.766707 14768 log.cc:1196] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/74852c49f5fc43b4a1a0388f124c607b
I20260620 07:00:39.767273 14768 ts_tablet_manager.cc:1939] T 74852c49f5fc43b4a1a0388f124c607b P cd265dde62314bdb8ebadcbcaa593dc0: Deleting consensus metadata
I20260620 07:00:39.769289 14252 catalog_manager.cc:4841] TS cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079): tablet 74852c49f5fc43b4a1a0388f124c607b (table test-table [id=863760e5c8a545538f83891c87717144]) successfully deleted
I20260620 07:00:39.769932 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.739356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.0.97.1:39275)
0620 07:00:39.739542 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.739549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.751108 (+ 11559us) server_negotiation.cc:407] Connection header received
0620 07:00:39.751306 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.751312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.751418 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.751514 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:39.753295 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.754318 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.755884 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.756486 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.756621 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.762328 (+ 5707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:39.762346 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:39.762365 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:39.762416 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:39.766413 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.767407 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.767417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.767420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.767590 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.768074 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.768079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.768081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.768539 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:39.768808 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.769386 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.769604 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
I20260620 07:00:39.770140 14766 ts_tablet_manager.cc:1918] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:39.770350 14766 log.cc:1196] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/74852c49f5fc43b4a1a0388f124c607b
I20260620 07:00:39.770905 14766 ts_tablet_manager.cc:1939] T 74852c49f5fc43b4a1a0388f124c607b P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting consensus metadata
I20260620 07:00:39.771204 14452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:39.772696 14254 catalog_manager.cc:4841] TS 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2:40469): tablet 74852c49f5fc43b4a1a0388f124c607b (table test-table [id=863760e5c8a545538f83891c87717144]) successfully deleted
I20260620 07:00:39.873790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14329
W20260620 07:00:39.888754 14253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39275 (error 108)
I20260620 07:00:39.888922 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14463
I20260620 07:00:39.891491 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.890911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37066)
0620 07:00:39.891183 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.891300 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260620 07:00:39.891731 14253 catalog_manager.cc:4584] TS d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74852c49f5fc43b4a1a0388f124c607b: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Jun 20 07:00:39 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:00:39.905097 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.889708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.97.2:40469)
0620 07:00:39.889904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:39.889914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:39.889958 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:00:39.890162 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:39.890168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:39.890261 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:39.890365 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:39.892099 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.892933 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.894251 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:39.894907 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:39.895021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:39.899107 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:39.899121 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:39.899135 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:39.899174 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:39.902208 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.903025 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.903039 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.903041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.903186 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:39.903539 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:39.903542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:39.903545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:39.903913 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:39.904237 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:39.904517 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:00:39.904785 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:00:39.905869 14586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:39.945328 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.944831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37082)
0620 07:00:39.945061 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.945164 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:00:39.946408 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:39.946049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37090)
0620 07:00:39.946203 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:39.946292 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:00:40.007266 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.006772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37096)
0620 07:00:40.006984 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:40.007078 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 07:00:40.008363 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.008006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37106)
0620 07:00:40.008161 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:40.008233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 07:00:40.020144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14463
I20260620 07:00:40.035106 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14596
Jun 20 07:00:40 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:00:40.052357 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.035831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.0.97.3:37079)
0620 07:00:40.036053 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:40.036058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:40.036078 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:00:40.036255 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:40.036260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:40.036357 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:40.036494 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:40.037731 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:40.038459 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:40.040145 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:40.040553 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:40.040660 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:40.045432 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:40.045450 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:40.045467 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:40.045511 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:40.048529 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:40.049333 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:40.049341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:40.049344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:40.049514 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:40.050111 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:40.050118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:40.050121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:40.050806 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:40.051063 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:40.051592 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 07:00:40.052064 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 07:00:40.053126 14719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:40.089211 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.088606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37118)
0620 07:00:40.088865 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:40.088981 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 07:00:40.090519 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.090047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37120)
0620 07:00:40.090272 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:40.090365 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 07:00:40.143877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14596
I20260620 07:00:40.157361 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14236
I20260620 07:00:40.158563 14298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:40.271852 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:40.271195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37134)
0620 07:00:40.271539 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:40.271659 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 07:00:40.271857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14236
I20260620 07:00:40.289633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--webserver_interface=127.0.97.62
--webserver_port=42231
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:40.613826 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:40.614137 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:40.614253 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:40.621711 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:40.621824 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:40.621882 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:40.621959 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:40.622012 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:40.632233 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:40.633612 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:40.635005 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:40.656893 14784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:40.656929 14782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:40.657205 14781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:40.657773 14775 server_base.cc:1034] running on GCE node
I20260620 07:00:40.659269 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:40.661006 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:40.662242 14775 hybrid_clock.cc:648] HybridClock initialized: now 1781938840662179 us; error 60 us; skew 500 ppm
I20260620 07:00:40.662717 14775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:40 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:40.666298 14775 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:40.668102 14775 webserver.cc:460] Webserver started at http://127.0.97.62:42231/ using document root <none> and password file <none>
I20260620 07:00:40.668877 14775 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:40.669018 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:40.675325 14775 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:00:40.678577 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:40.680073 14775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:00:40.680290 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "20ba7a9e8fe2464588420cdb5078754e"
format_stamp: "Formatted at 2026-06-20 07:00:37 on dist-test-slave-mz20"
server_key: "99b2627f8e267fb58c0da3bd1be68c24"
server_key_iv: "78dc9cd93a2d3ddf0ea4c91aa36b6093"
server_key_version: "encryptionkey@0"
I20260620 07:00:40.681418 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:40.704550 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:40.708700 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:40.708964 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:40.726846 14775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37161
I20260620 07:00:40.726873 14842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37161 every 8 connection(s)
I20260620 07:00:40.728413 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:40.731841 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14775
I20260620 07:00:40.732769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39275
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=36911
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:40.737075 14843 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:00:40.741488 14843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap starting.
I20260620 07:00:40.761396 14843 log.cc:824] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:40.773440 14843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:00:40.773988 14843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap complete.
I20260620 07:00:40.783025 14843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:40.783877 14843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Initialized, Role: FOLLOWER
I20260620 07:00:40.784520 14843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:40.784742 14843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:40.784859 14843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:40.785020 14843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:00:40.789017 14843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:40.789510 14843 leader_election.cc:302] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ba7a9e8fe2464588420cdb5078754e; no voters:
I20260620 07:00:40.790148 14843 leader_election.cc:288] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:00:40.790275 14847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:00:40.792449 14843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:40.794808 14847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 LEADER]: Becoming Leader. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Running, Role: LEADER
I20260620 07:00:40.795504 14847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:40.798919 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ba7a9e8fe2464588420cdb5078754e. Latest consensus state: current_term: 2 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:40.799335 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:40.804312 14848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:40.804697 14848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:40.810176 14854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:40.819871 14854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:40.821412 14854 catalog_manager.cc:1259] Loaded cluster ID: 5522f388de90430b81e2b02247bde665
I20260620 07:00:40.821589 14854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:00:40.824692 14854 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 07:00:40.841339 14854 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 07:00:40.850885 14854 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:00:40.852604 14854 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Loaded TSK: 0
I20260620 07:00:40.854235 14854 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:00:41.093415 14845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:41.093896 14845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:41.094131 14845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:41.101657 14845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:41.101788 14845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:41.101939 14845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:00:41.112175 14845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36911
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:41.113458 14845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:41.114872 14845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:41.129493 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:41.129808 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:41.129914 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:41.131523 14845 server_base.cc:1034] running on GCE node
I20260620 07:00:41.132237 14845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:41.133338 14845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:41.134593 14845 hybrid_clock.cc:648] HybridClock initialized: now 1781938841134499 us; error 73 us; skew 500 ppm
I20260620 07:00:41.134938 14845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:41 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:41.138209 14845 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:00:41.139885 14845 webserver.cc:460] Webserver started at http://127.0.97.1:36911/ using document root <none> and password file <none>
I20260620 07:00:41.140635 14845 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:41.140759 14845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:41.146344 14845 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:00:41.149494 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:41.151042 14845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:41.151217 14845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d91e9fbfadc44ef2870569b65cd7736f"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "084ce8c88d85567ffc60abd17df63ad7"
server_key_iv: "86ee374d8261475fbba4a6f3a194559d"
server_key_version: "encryptionkey@0"
I20260620 07:00:41.152251 14845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:41.183064 14845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:41.187503 14845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:41.187758 14845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:41.188779 14845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:41.190714 14887 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 07:00:41.196184 14887 ts_tablet_manager.cc:1852] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 07:00:41.197261 14887 ts_tablet_manager.cc:1905] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:41.204553 14887 ts_tablet_manager.cc:1918] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:41.204756 14887 log.cc:1196] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/74852c49f5fc43b4a1a0388f124c607b
I20260620 07:00:41.205204 14887 ts_tablet_manager.cc:1939] T 74852c49f5fc43b4a1a0388f124c607b P d91e9fbfadc44ef2870569b65cd7736f: Deleting consensus metadata
I20260620 07:00:41.205560 14845 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 07:00:41.205665 14845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260620 07:00:41.205780 14845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:41.205847 14845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:41.234967 14845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39275
I20260620 07:00:41.235035 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39275 every 8 connection(s)
I20260620 07:00:41.236195 14845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:00:41.238052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14845
I20260620 07:00:41.238660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40469
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=43779
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:41 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:41.261132 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:41.239264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:35225)
0620 07:00:41.240113 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:41.240168 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:41.242395 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:41.243820 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:41.243831 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:41.244526 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:41.245249 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:41.245283 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.247315 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.247319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:41.248540 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:41.248560 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.249820 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.249824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:41.250001 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:41.250651 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:41.250674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:41.252819 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:41.257133 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:41.257139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:41.257151 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:41.257579 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:41.258057 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:41.258060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:41.258062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:41.258231 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:41.259183 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:41.259188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:41.259510 (+ 322us) client_negotiation.cc:769] Sending connection context
0620 07:00:41.259739 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 07:00:41.260132 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":325,"threads_started":1}
I20260620 07:00:41.261716 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:41.238959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35225 (local address 127.0.97.62:37161)
0620 07:00:41.239592 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:41.239611 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:41.240310 (+ 699us) server_negotiation.cc:407] Connection header received
0620 07:00:41.242942 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:41.242956 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:41.243330 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:41.243629 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:41.245476 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.247135 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.248681 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.249634 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.249803 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:41.253072 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:41.253083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:41.253101 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:41.253147 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:41.256984 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:41.257711 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:41.257722 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:41.257730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:41.257956 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:41.258341 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:41.258346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:41.258349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:41.259048 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:41.259277 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:41.259995 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 07:00:41.260467 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260620 07:00:41.264568 14994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:41.264897 14994 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:41.265717 14994 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
I20260620 07:00:41.268079 14808 ts_manager.cc:194] Registered new tserver with Master: d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275)
I20260620 07:00:41.270118 14808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35225
W20260620 07:00:41.565753 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:41.566041 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:41.566135 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:41.573639 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:41.573769 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:41.573890 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:00:41.583815 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43779
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:41.585071 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:41.586467 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:41.599287 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:41.599328 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:41.599262 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:41.600041 14998 server_base.cc:1034] running on GCE node
I20260620 07:00:41.600610 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:41.601560 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:41.602759 14998 hybrid_clock.cc:648] HybridClock initialized: now 1781938841602705 us; error 38 us; skew 500 ppm
I20260620 07:00:41.603063 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:41 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:41.606120 14998 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:00:41.607666 14998 webserver.cc:460] Webserver started at http://127.0.97.2:43779/ using document root <none> and password file <none>
I20260620 07:00:41.608347 14998 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:41.608477 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:41.614288 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 07:00:41.617292 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:41.618863 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:00:41.619035 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "7af97e8a5a67424b9cb6916a9df7b8cf"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "637e8121f458b28af3352580a79b9427"
server_key_iv: "e0cc3c9c2fda6b38370bf263e242bd9c"
server_key_version: "encryptionkey@0"
I20260620 07:00:41.620023 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:41.641739 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:41.645844 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:41.646100 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:41.647118 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:41.648509 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:41.648626 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:41.648738 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:41.648808 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:41.679716 14998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40469
I20260620 07:00:41.679709 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40469 every 8 connection(s)
I20260620 07:00:41.681154 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:00:41.682782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 14998
I20260620 07:00:41.683343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37079
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=43275
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:00:41 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:41.707964 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:41.683873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59389 (local address 127.0.97.62:37161)
0620 07:00:41.684079 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:41.684087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:41.690068 (+ 5981us) server_negotiation.cc:407] Connection header received
0620 07:00:41.692195 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:41.692201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:41.692295 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:41.692409 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:41.694232 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.695245 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.696814 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.697402 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.697514 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:41.701073 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:41.701080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:41.701085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:41.701130 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:41.704472 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:41.705298 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:41.705304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:41.705307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:41.705482 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:41.706104 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:41.706109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:41.706112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:41.706489 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:41.706696 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:41.707341 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 07:00:41.707598 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:00:41.708760 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:41.684397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:59389)
0620 07:00:41.689828 (+ 5431us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:41.689911 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:41.691892 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:41.692740 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:41.692755 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:41.693387 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:41.694061 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:41.694081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.695423 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.695428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:41.696666 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:41.696678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:41.697615 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:41.697620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:41.697864 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:41.698515 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:41.698571 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:41.700828 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:41.704673 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:41.704684 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:41.704704 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:41.705171 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:41.705798 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:41.705801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:41.705803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:41.705990 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:41.706646 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:41.706651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:41.706977 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 07:00:41.707209 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 07:00:41.707611 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5068,"thread_start_us":4846,"threads_started":1}
I20260620 07:00:41.710150 15126 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:41.710425 15126 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:41.711225 15126 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
I20260620 07:00:41.712641 14808 ts_manager.cc:194] Registered new tserver with Master: 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2:40469)
I20260620 07:00:41.713845 14808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59389
W20260620 07:00:42.007176 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:42.007447 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:42.007551 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:42.014787 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:42.014899 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:42.015017 15129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:00:42.024600 15129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43275
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:42.025806 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:42.027066 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:42.039393 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:42.039474 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:42.039435 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:42.040314 15129 server_base.cc:1034] running on GCE node
I20260620 07:00:42.040899 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:42.041790 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:42.042999 15129 hybrid_clock.cc:648] HybridClock initialized: now 1781938842042950 us; error 41 us; skew 500 ppm
I20260620 07:00:42.043304 15129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:42 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:42.046288 15129 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:00:42.047811 15129 webserver.cc:460] Webserver started at http://127.0.97.3:43275/ using document root <none> and password file <none>
I20260620 07:00:42.048501 15129 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:42.048631 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:42.054126 15129 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:00:42.057091 15144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:42.058502 15129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:00:42.058678 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
format_stamp: "Formatted at 2026-06-20 07:00:39 on dist-test-slave-mz20"
server_key: "b3168cf00da08004c7a898502809fac6"
server_key_iv: "691186771a08c1bd8ae4a7bf933daeb7"
server_key_version: "encryptionkey@0"
I20260620 07:00:42.059635 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:42.072213 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:42.075906 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:42.076123 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:42.077044 15129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:42.078384 15129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:42.078469 15129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:42.078581 15129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:42.078650 15129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 07:00:42.110240 15129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37079
I20260620 07:00:42.110318 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37079 every 8 connection(s)
I20260620 07:00:42.111706 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:00:42.118036 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15129
Jun 20 07:00:42 dist-test-slave-mz20 krb5kdc[14223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:00:42.132738 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.114224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33167 (local address 127.0.97.62:37161)
0620 07:00:42.114429 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.114435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.115662 (+ 1227us) server_negotiation.cc:407] Connection header received
0620 07:00:42.117411 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.117415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.117485 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.117573 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:42.119595 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.120576 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.122286 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.122740 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.122873 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.126582 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:42.126590 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:42.126596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:42.126636 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:42.129509 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.130345 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.130349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.130352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.130473 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.130933 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.130938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.130940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.131282 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:42.131412 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.132197 (+ 785us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.132415 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:00:42.133428 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.119000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.97.62:37161)
0620 07:00:42.119911 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.119915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.119929 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:00:42.119967 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.119970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.120023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.120100 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:42.121707 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.122684 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.124141 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.124786 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.124924 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.127686 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:42.127691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:42.127694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:42.127732 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:42.130630 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.131340 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.131346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.131349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.131494 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.131825 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.131829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.131832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.132218 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:42.132447 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.132772 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.133064 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"spinlock_wait_cycles":5504,"thread_start_us":696,"threads_started":1}
I20260620 07:00:42.133999 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.114784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:33167)
0620 07:00:42.115495 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.115539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.117153 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.117846 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.117858 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.118625 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:00:42.119394 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.119422 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.120725 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.120729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.122143 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.122158 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.122968 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.122974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.123172 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.123825 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:00:42.123848 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:00:42.126322 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:00:42.129637 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:42.129645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:42.129663 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:42.130222 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:42.130599 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:00:42.130604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:00:42.130608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:00:42.130833 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:00:42.131405 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:00:42.131413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:00:42.131779 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.132085 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.132601 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":271,"threads_started":1}
I20260620 07:00:42.135268 15257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37161
I20260620 07:00:42.135597 15257 heartbeater.cc:464] Registering TS with master...
I20260620 07:00:42.136241 15257 heartbeater.cc:511] Master 127.0.97.62:37161 requested a full tablet report, sending...
I20260620 07:00:42.137480 14808 ts_manager.cc:194] Registered new tserver with Master: cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079)
I20260620 07:00:42.138244 14808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33167
I20260620 07:00:42.148177 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:00:42.154990 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.149030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.0.97.62:37161)
0620 07:00:42.149269 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.149274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.150087 (+ 813us) server_negotiation.cc:407] Connection header received
0620 07:00:42.150251 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.150255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.150398 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.150526 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:00:42.150914 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.151596 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.153055 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.153480 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.153572 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.153980 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:00:42.154470 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:00:42.154618 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.154719 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.154798 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 07:00:42.155993 14808 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:00:42.163547 14808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:00:42.196745 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.175208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.97.3:37079)
0620 07:00:42.188982 (+ 13774us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.188989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.189022 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:00:42.189126 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.189134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.189416 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.189609 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:42.190244 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.191775 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.194523 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.195661 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.195777 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.196017 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.196260 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.196426 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13609,"thread_start_us":13443,"threads_started":1}
I20260620 07:00:42.197141 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.174940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37079 (local address 127.0.0.1:47840)
0620 07:00:42.186557 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.186609 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.186847 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.189628 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.189635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.189659 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:42.190011 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.190024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.191923 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.191928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.194336 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.194351 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.195857 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.195863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.196037 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.196085 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.196203 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.196389 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11256,"mutex_wait_us":105,"thread_start_us":14111,"threads_started":1}
I20260620 07:00:42.197487 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.176249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.97.2:40469)
0620 07:00:42.185865 (+ 9616us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.185877 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.189561 (+ 3684us) server_negotiation.cc:407] Connection header received
0620 07:00:42.189664 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.189673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.189905 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.190062 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:42.190731 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.191908 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.194803 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.196134 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.196254 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.196471 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.197019 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.197172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9390,"thread_start_us":9255,"threads_started":1}
I20260620 07:00:42.197487 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.175561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40469 (local address 127.0.0.1:34442)
0620 07:00:42.189324 (+ 13763us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.189341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.189475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.190078 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.190082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.190105 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:42.190515 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.190529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.192052 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.192056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.194314 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.194328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.196627 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.196634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.196827 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.196862 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.196946 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.197063 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13641,"mutex_wait_us":65,"thread_start_us":10512,"threads_started":1}
I20260620 07:00:42.197901 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.175334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37146)
0620 07:00:42.189053 (+ 13719us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.189073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.189204 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.190753 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.190757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.190781 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:42.191119 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.191132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.192857 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.192863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.194964 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.194979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.197153 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.197164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.197351 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.197385 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.197467 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.197591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13581,"thread_start_us":13966,"threads_started":1}
I20260620 07:00:42.198653 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.175905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.97.1:39275)
0620 07:00:42.188429 (+ 12524us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.188439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.189317 (+ 878us) server_negotiation.cc:407] Connection header received
0620 07:00:42.189430 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.189438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.189681 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.190478 (+ 797us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:42.191315 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.192712 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.195121 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.196465 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.196586 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.196751 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.198150 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.198323 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12325,"thread_start_us":12535,"threads_started":1}
I20260620 07:00:42.200616 14929 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5e5801f9547618660a4c6a88908fb (DEFAULT_TABLE table=test-table [id=9337d80872c3472db609110e9a81ff9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:42.201058 15061 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5e5801f9547618660a4c6a88908fb (DEFAULT_TABLE table=test-table [id=9337d80872c3472db609110e9a81ff9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:42.201141 15192 tablet_service.cc:1467] Processing CreateTablet for tablet a8e5e5801f9547618660a4c6a88908fb (DEFAULT_TABLE table=test-table [id=9337d80872c3472db609110e9a81ff9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:00:42.201972 14929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5e5801f9547618660a4c6a88908fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:42.202050 15061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5e5801f9547618660a4c6a88908fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:42.202391 15192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8e5e5801f9547618660a4c6a88908fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:00:42.212009 15273 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Bootstrap starting.
I20260620 07:00:42.215116 15274 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Bootstrap starting.
I20260620 07:00:42.215862 15273 tablet_bootstrap.cc:654] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:42.217355 15273 log.cc:824] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:42.218616 15274 tablet_bootstrap.cc:654] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:42.219693 15274 log.cc:824] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:42.220196 15273 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: No bootstrap required, opened a new log
I20260620 07:00:42.220419 15273 ts_tablet_manager.cc:1397] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:00:42.222007 15274 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: No bootstrap required, opened a new log
I20260620 07:00:42.222223 15274 ts_tablet_manager.cc:1397] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 07:00:42.222640 15275 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Bootstrap starting.
I20260620 07:00:42.226243 15275 tablet_bootstrap.cc:654] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Neither blocks nor log segments found. Creating new log.
I20260620 07:00:42.227612 15275 log.cc:824] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:42.228083 15273 raft_consensus.cc:348] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.228546 15273 raft_consensus.cc:374] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:42.228654 15273 raft_consensus.cc:729] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91e9fbfadc44ef2870569b65cd7736f, State: Initialized, Role: FOLLOWER
I20260620 07:00:42.229274 15273 consensus_queue.cc:260] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.230239 14994 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:42.230934 15273 ts_tablet_manager.cc:1428] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260620 07:00:42.230973 15275 tablet_bootstrap.cc:492] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: No bootstrap required, opened a new log
I20260620 07:00:42.231227 15275 ts_tablet_manager.cc:1397] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260620 07:00:42.231895 15274 raft_consensus.cc:348] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.232347 15274 raft_consensus.cc:374] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:42.232451 15274 raft_consensus.cc:729] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af97e8a5a67424b9cb6916a9df7b8cf, State: Initialized, Role: FOLLOWER
I20260620 07:00:42.233032 15274 consensus_queue.cc:260] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.234143 15126 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:42.234697 15274 ts_tablet_manager.cc:1428] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
W20260620 07:00:42.238135 14995 tablet.cc:2369] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:42.243600 15275 raft_consensus.cc:348] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.243942 15275 raft_consensus.cc:374] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:00:42.244033 15275 raft_consensus.cc:729] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd265dde62314bdb8ebadcbcaa593dc0, State: Initialized, Role: FOLLOWER
I20260620 07:00:42.244534 15275 consensus_queue.cc:260] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.245587 15257 heartbeater.cc:503] Master 127.0.97.62:37161 was elected leader, sending a full tablet report...
I20260620 07:00:42.246031 15275 ts_tablet_manager.cc:1428] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Time spent starting tablet: real 0.015s user 0.009s sys 0.001s
W20260620 07:00:42.363263 15258 tablet.cc:2369] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:00:42.433125 15127 tablet.cc:2369] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:00:42.637543 15280 raft_consensus.cc:482] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:00:42.637962 15280 raft_consensus.cc:504] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.640375 15280 leader_election.cc:288] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079), d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275)
I20260620 07:00:42.647742 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.641011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56945 (local address 127.0.97.1:39275)
0620 07:00:42.641384 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.641389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.641973 (+ 584us) server_negotiation.cc:407] Connection header received
0620 07:00:42.642140 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.642144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.642218 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.642301 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:42.643003 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.643800 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.645851 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.646704 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.646801 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.646911 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.647301 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.647449 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260620 07:00:42.647742 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.640919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42981 (local address 127.0.97.3:37079)
0620 07:00:42.641169 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.641177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.641853 (+ 676us) server_negotiation.cc:407] Connection header received
0620 07:00:42.642111 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.642118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.642197 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.642273 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:00:42.643060 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.643796 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.645869 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.646749 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.646880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.646988 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.647299 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.647441 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 07:00:42.647909 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.640582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37079 (local address 127.0.97.2:42981)
0620 07:00:42.641720 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.641745 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.641922 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.642305 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.642310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.642334 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:42.642906 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.642917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.643918 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.643923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.645722 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.645733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.646969 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.646973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.647158 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.647181 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.647259 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.647455 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":177,"thread_start_us":1007,"threads_started":1}
I20260620 07:00:42.648247 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.640800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.97.2:56945)
0620 07:00:42.641876 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.641890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:00:42.642014 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:00:42.642324 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:00:42.642328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:00:42.642350 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:00:42.642838 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.642851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.643932 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.643937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.645702 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:00:42.645717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.646937 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.646943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:00:42.647124 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.647175 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 07:00:42.647259 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:00:42.647455 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":978,"mutex_wait_us":155,"thread_start_us":454,"threads_started":1}
I20260620 07:00:42.649087 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5e5801f9547618660a4c6a88908fb" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" is_pre_election: true
I20260620 07:00:42.649245 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5e5801f9547618660a4c6a88908fb" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91e9fbfadc44ef2870569b65cd7736f" is_pre_election: true
I20260620 07:00:42.649654 15212 raft_consensus.cc:2454] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 0.
I20260620 07:00:42.649746 14949 raft_consensus.cc:2454] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 0.
I20260620 07:00:42.650348 15014 leader_election.cc:302] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7af97e8a5a67424b9cb6916a9df7b8cf, cd265dde62314bdb8ebadcbcaa593dc0, d91e9fbfadc44ef2870569b65cd7736f; no voters:
I20260620 07:00:42.650833 15280 raft_consensus.cc:2781] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:00:42.650985 15280 raft_consensus.cc:482] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:00:42.651108 15280 raft_consensus.cc:3037] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:42.653753 15280 raft_consensus.cc:504] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.654470 15280 leader_election.cc:288] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 election: Requested vote from peers cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079), d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275)
I20260620 07:00:42.655067 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5e5801f9547618660a4c6a88908fb" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91e9fbfadc44ef2870569b65cd7736f"
I20260620 07:00:42.655038 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8e5e5801f9547618660a4c6a88908fb" candidate_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
I20260620 07:00:42.655300 14949 raft_consensus.cc:3037] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:42.655300 15212 raft_consensus.cc:3037] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:00:42.658906 15212 raft_consensus.cc:2454] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 1.
I20260620 07:00:42.658912 14949 raft_consensus.cc:2454] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af97e8a5a67424b9cb6916a9df7b8cf in term 1.
I20260620 07:00:42.659375 15015 leader_election.cc:302] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7af97e8a5a67424b9cb6916a9df7b8cf, d91e9fbfadc44ef2870569b65cd7736f; no voters:
I20260620 07:00:42.659711 15280 raft_consensus.cc:2781] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:00:42.660280 15280 raft_consensus.cc:686] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 LEADER]: Becoming Leader. State: Replica: 7af97e8a5a67424b9cb6916a9df7b8cf, State: Running, Role: LEADER
I20260620 07:00:42.660915 15280 consensus_queue.cc:237] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } }
I20260620 07:00:42.666469 14806 catalog_manager.cc:5495] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af97e8a5a67424b9cb6916a9df7b8cf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40469 } health_report { overall_health: HEALTHY } } }
I20260620 07:00:42.764361 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.757814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.0.97.2:40469)
0620 07:00:42.758598 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.758603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.758622 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:00:42.758732 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.758736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.758837 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.758993 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:00:42.759396 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.760157 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.761745 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.762221 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.762335 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.762983 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:00:42.763696 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:00:42.763825 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.763935 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.764013 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":519,"threads_started":1}
I20260620 07:00:42.773718 14949 raft_consensus.cc:1261] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Refusing update from remote peer 7af97e8a5a67424b9cb6916a9df7b8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:42.773710 15212 raft_consensus.cc:1261] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Refusing update from remote peer 7af97e8a5a67424b9cb6916a9df7b8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:00:42.774740 15284 consensus_queue.cc:1035] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd265dde62314bdb8ebadcbcaa593dc0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:42.775125 15280 consensus_queue.cc:1035] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91e9fbfadc44ef2870569b65cd7736f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:00:42.787817 15288 mvcc.cc:205] Tried to move back new op lower bound from 7298821499990392832 to 7298821499543846912. Current Snapshot: MvccSnapshot[applied={T|T < 7298821499990392832}]
I20260620 07:00:42.791543 15292 mvcc.cc:205] Tried to move back new op lower bound from 7298821499990392832 to 7298821499543846912. Current Snapshot: MvccSnapshot[applied={T|T < 7298821499990392832}]
I20260620 07:00:42.802372 14808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:42.802687 14808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:00:42.806403 14808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Sending DeleteTablet for 3 replicas of tablet a8e5e5801f9547618660a4c6a88908fb
I20260620 07:00:42.807756 15192 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e5e5801f9547618660a4c6a88908fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47840
I20260620 07:00:42.807801 14929 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e5e5801f9547618660a4c6a88908fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37146
I20260620 07:00:42.807981 15061 tablet_service.cc:1514] Processing DeleteTablet for tablet a8e5e5801f9547618660a4c6a88908fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:00:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34442
I20260620 07:00:42.808429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14845
I20260620 07:00:42.823175 15300 tablet_replica.cc:330] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: stopping tablet replica
I20260620 07:00:42.824548 15300 raft_consensus.cc:2229] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:42.822479 15299 tablet_replica.cc:330] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: stopping tablet replica
I20260620 07:00:42.825456 15299 raft_consensus.cc:2229] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:00:42.826100 15299 raft_consensus.cc:2258] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:42.826894 15300 raft_consensus.cc:2258] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:42.829198 15300 ts_tablet_manager.cc:1905] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:42.832710 15301 tablet_replica.cc:330] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: stopping tablet replica
I20260620 07:00:42.833248 15301 raft_consensus.cc:2229] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:00:42.833913 15299 ts_tablet_manager.cc:1905] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:42.834067 15301 raft_consensus.cc:2258] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:00:42.836246 15301 ts_tablet_manager.cc:1905] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:42.839548 15300 ts_tablet_manager.cc:1918] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:42.839803 15300 log.cc:1196] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/a8e5e5801f9547618660a4c6a88908fb
I20260620 07:00:42.840523 15300 ts_tablet_manager.cc:1939] T a8e5e5801f9547618660a4c6a88908fb P cd265dde62314bdb8ebadcbcaa593dc0: Deleting consensus metadata
I20260620 07:00:42.842972 14792 catalog_manager.cc:4841] TS cd265dde62314bdb8ebadcbcaa593dc0 (127.0.97.3:37079): tablet a8e5e5801f9547618660a4c6a88908fb (table test-table [id=9337d80872c3472db609110e9a81ff9f]) successfully deleted
I20260620 07:00:42.843333 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.809374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.97.1:39275)
0620 07:00:42.809579 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.809587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.823973 (+ 14386us) server_negotiation.cc:407] Connection header received
0620 07:00:42.824290 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.824298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.824398 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.824487 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:42.827216 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.828265 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.830284 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.830941 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.831037 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:42.835036 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:42.835052 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:42.835073 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:42.835126 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:42.839487 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.840498 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.840510 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.840515 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.840698 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:42.841188 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:42.841193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:42.841195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:42.841688 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:42.842013 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:42.842654 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 07:00:42.842948 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:00:42.843858 15301 ts_tablet_manager.cc:1918] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:42.844162 15301 log.cc:1196] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/a8e5e5801f9547618660a4c6a88908fb
I20260620 07:00:42.844882 15301 ts_tablet_manager.cc:1939] T a8e5e5801f9547618660a4c6a88908fb P 7af97e8a5a67424b9cb6916a9df7b8cf: Deleting consensus metadata
I20260620 07:00:42.844950 14989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:42.847707 14794 catalog_manager.cc:4841] TS 7af97e8a5a67424b9cb6916a9df7b8cf (127.0.97.2:40469): tablet a8e5e5801f9547618660a4c6a88908fb (table test-table [id=9337d80872c3472db609110e9a81ff9f]) successfully deleted
I20260620 07:00:42.973058 15299 ts_tablet_manager.cc:1918] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:42.973335 15299 log.cc:1196] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/a8e5e5801f9547618660a4c6a88908fb
I20260620 07:00:42.973806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14845
I20260620 07:00:42.974000 15299 ts_tablet_manager.cc:1939] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Deleting consensus metadata
I20260620 07:00:42.991935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14998
W20260620 07:00:42.992177 14793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39275 (error 108)
I20260620 07:00:42.994877 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.993744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37154)
0620 07:00:42.994410 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:42.994576 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":377,"threads_started":1}
W20260620 07:00:42.995190 14793 catalog_manager.cc:4584] TS d91e9fbfadc44ef2870569b65cd7736f (127.0.97.1:39275): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8e5e5801f9547618660a4c6a88908fb: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
I20260620 07:00:43.010133 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:42.992904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.97.2:40469)
0620 07:00:42.993205 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:42.993212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:42.993487 (+ 275us) server_negotiation.cc:407] Connection header received
0620 07:00:42.993586 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:42.993592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:42.993686 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:42.993839 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:42.995708 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.997014 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.998422 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:42.999192 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:42.999349 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:43.002472 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:43.002495 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:43.002522 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:43.002598 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:43.006819 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:43.007621 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:43.007633 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:43.007637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:43.007807 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:43.008212 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:43.008218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:43.008221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:43.008769 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:43.009122 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:43.009409 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 07:00:43.009682 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260620 07:00:43.011195 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:43.049135 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.048480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37160)
0620 07:00:43.048775 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.048908 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 07:00:43.050509 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.050042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37168)
0620 07:00:43.050225 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.050320 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 07:00:43.111773 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.111055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37182)
0620 07:00:43.111382 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.111509 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 07:00:43.113116 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.112686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37186)
0620 07:00:43.112852 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.112943 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:00:43.137954 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14998
I20260620 07:00:43.156615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15129
I20260620 07:00:43.176494 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.162184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.97.3:37079)
0620 07:00:43.162832 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 07:00:43.162837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:00:43.162855 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:00:43.162923 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:00:43.162928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:00:43.162997 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:00:43.163131 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:00:43.164741 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:43.165644 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:43.167150 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:00:43.167579 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:00:43.167694 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:00:43.170457 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:00:43.170475 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:00:43.170496 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:00:43.170535 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:00:43.173528 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:43.174252 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:43.174261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:43.174264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:43.174416 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:00:43.174831 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:00:43.174834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:00:43.174837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:00:43.175251 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:00:43.175570 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 07:00:43.175864 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:00:43.176100 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":339,"threads_started":1}
I20260620 07:00:43.177282 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:43.194144 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.193407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37196)
0620 07:00:43.193731 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.193897 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:00:43.195588 15305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:43.195015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39275 (local address 127.0.0.1:37212)
0620 07:00:43.195287 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:43.195394 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 07:00:43.274102 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15129
I20260620 07:00:43.289996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 14775
I20260620 07:00:43.291460 14838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:00:43.420575 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 14775
I20260620 07:00:43.443226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--webserver_interface=127.0.97.62
--webserver_port=42231
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:43.819936 15310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:43.820281 15310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:43.820428 15310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:43.828415 15310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:00:43.828552 15310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:43.828624 15310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:43.828709 15310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:00:43.828779 15310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:00:43.840381 15310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:43.842145 15310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:43.843899 15310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:43.856768 15315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:43.856796 15316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:43.856791 15318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:43.857647 15310 server_base.cc:1034] running on GCE node
I20260620 07:00:43.858608 15310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:43.860285 15310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:43.861491 15310 hybrid_clock.cc:648] HybridClock initialized: now 1781938843861438 us; error 42 us; skew 500 ppm
I20260620 07:00:43.861889 15310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:43 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:43.865461 15310 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:00:43.867168 15310 webserver.cc:460] Webserver started at http://127.0.97.62:42231/ using document root <none> and password file <none>
I20260620 07:00:43.867959 15310 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:43.868086 15310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:43.874331 15310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:00:43.877408 15325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:43.878924 15310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:43.879081 15310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "20ba7a9e8fe2464588420cdb5078754e"
format_stamp: "Formatted at 2026-06-20 07:00:37 on dist-test-slave-mz20"
server_key: "99b2627f8e267fb58c0da3bd1be68c24"
server_key_iv: "78dc9cd93a2d3ddf0ea4c91aa36b6093"
server_key_version: "encryptionkey@0"
I20260620 07:00:43.880131 15310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:43.906448 15310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:43.911509 15310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:43.911844 15310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:43.932507 15310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37161
I20260620 07:00:43.932619 15376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37161 every 8 connection(s)
I20260620 07:00:43.934527 15310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:00:43.941061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15310
I20260620 07:00:43.942119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39275
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=36911
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:43.943554 15377 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:00:43.948648 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap starting.
I20260620 07:00:43.973260 15377 log.cc:824] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:00:43.991324 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:00:43.991918 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e: Bootstrap complete.
I20260620 07:00:44.001626 15377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:44.002648 15377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Initialized, Role: FOLLOWER
I20260620 07:00:44.003427 15377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:44.003683 15377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:00:44.003803 15377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:00:44.003985 15377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:00:44.008383 15377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:44.008929 15377 leader_election.cc:302] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ba7a9e8fe2464588420cdb5078754e; no voters:
I20260620 07:00:44.009640 15377 leader_election.cc:288] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:00:44.009799 15381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:00:44.010834 15381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [term 3 LEADER]: Becoming Leader. State: Replica: 20ba7a9e8fe2464588420cdb5078754e, State: Running, Role: LEADER
I20260620 07:00:44.011478 15381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } }
I20260620 07:00:44.011970 15377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:00:44.014331 15383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ba7a9e8fe2464588420cdb5078754e. Latest consensus state: current_term: 3 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:44.014421 15382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "20ba7a9e8fe2464588420cdb5078754e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ba7a9e8fe2464588420cdb5078754e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37161 } } }
I20260620 07:00:44.014708 15383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:44.014782 15382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ba7a9e8fe2464588420cdb5078754e [sys.catalog]: This master's current role is: LEADER
I20260620 07:00:44.015600 15387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:00:44.036331 15387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:00:44.038532 15387 catalog_manager.cc:1259] Loaded cluster ID: 5522f388de90430b81e2b02247bde665
I20260620 07:00:44.038779 15387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 07:00:44.041235 15387 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff6129639ad google::LogMessage::Fail() at ??:0
@ 0x7ff61296688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff6129634de google::LogMessage::Flush() at ??:0
@ 0x7ff612964a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff628396512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff628393976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff62841b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff62841b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff6155af345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff6155b291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff6155b2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff61558deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6188676db start_thread at ??:0
@ 0x7ff6117fd61f clone at ??:0
W20260620 07:00:44.363114 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:44.363539 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:44.363703 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:44.372108 15379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:44.372274 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:44.372452 15379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:00:44.383723 15379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36911
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:44.385506 15379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:44.387153 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:44.402287 15417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:44.403177 15418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:44.403648 15420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:44.404500 15379 server_base.cc:1034] running on GCE node
I20260620 07:00:44.405181 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:44.406275 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:44.407516 15379 hybrid_clock.cc:648] HybridClock initialized: now 1781938844407438 us; error 61 us; skew 500 ppm
I20260620 07:00:44.407874 15379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:44 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:44.411561 15379 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:00:44.413287 15379 webserver.cc:460] Webserver started at http://127.0.97.1:36911/ using document root <none> and password file <none>
I20260620 07:00:44.414165 15379 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:44.414292 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:44.420609 15379 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:00:44.423835 15427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:44.425519 15379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:00:44.425704 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d91e9fbfadc44ef2870569b65cd7736f"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "084ce8c88d85567ffc60abd17df63ad7"
server_key_iv: "86ee374d8261475fbba4a6f3a194559d"
server_key_version: "encryptionkey@0"
I20260620 07:00:44.426997 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:44.443986 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:44.448843 15379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:44.449153 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:44.450378 15379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:44.452518 15434 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 07:00:44.459178 15434 ts_tablet_manager.cc:1852] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 07:00:44.460335 15434 ts_tablet_manager.cc:1905] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:00:44.469485 15434 ts_tablet_manager.cc:1918] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:00:44.469825 15434 ts_tablet_manager.cc:1939] T a8e5e5801f9547618660a4c6a88908fb P d91e9fbfadc44ef2870569b65cd7736f: Deleting consensus metadata
I20260620 07:00:44.470381 15379 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 07:00:44.470526 15379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260620 07:00:44.470736 15379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:44.470817 15379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:44.506052 15379 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39275
I20260620 07:00:44.506088 15540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39275 every 8 connection(s)
I20260620 07:00:44.507694 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:00:44.513743 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:44.511438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:45265)
0620 07:00:44.512430 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:44.512674 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":357,"threads_started":1}
I20260620 07:00:44.513840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15379
W20260620 07:00:44.514642 15541 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37161: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
I20260620 07:00:44.514688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40469
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=43779
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:00:44.515676 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:44.515271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:47257)
0620 07:00:44.515477 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:44.515557 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:00:44.516613 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:44.516255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:50003)
0620 07:00:44.516404 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:44.516480 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:00:44.516830 15541 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 07:00:44.877741 15544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:44.878036 15544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:44.878129 15544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:44.885488 15544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:44.885612 15544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:44.885730 15544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:00:44.895682 15544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43779
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:44.897004 15544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:44.898388 15544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:44.911957 15549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:44.911927 15550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:44.912037 15552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:44.912726 15544 server_base.cc:1034] running on GCE node
I20260620 07:00:44.913388 15544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:44.914501 15544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:44.915750 15544 hybrid_clock.cc:648] HybridClock initialized: now 1781938844915696 us; error 47 us; skew 500 ppm
I20260620 07:00:44.916105 15544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:44 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:44.919401 15544 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:00:44.920915 15544 webserver.cc:460] Webserver started at http://127.0.97.2:43779/ using document root <none> and password file <none>
I20260620 07:00:44.921618 15544 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:44.921736 15544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:44.927260 15544 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 07:00:44.930055 15559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:44.931562 15544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:00:44.931708 15544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "7af97e8a5a67424b9cb6916a9df7b8cf"
format_stamp: "Formatted at 2026-06-20 07:00:38 on dist-test-slave-mz20"
server_key: "637e8121f458b28af3352580a79b9427"
server_key_iv: "e0cc3c9c2fda6b38370bf263e242bd9c"
server_key_version: "encryptionkey@0"
I20260620 07:00:44.932727 15544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:44.953687 15544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:44.957700 15544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:44.957962 15544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:44.958973 15544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:44.960354 15544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:44.960456 15544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:44.960564 15544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:44.960632 15544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:44.991979 15544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40469
I20260620 07:00:44.992035 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40469 every 8 connection(s)
I20260620 07:00:44.993428 15544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:00:44.998392 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:44.996403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:40091)
0620 07:00:44.997131 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:44.997350 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":286,"threads_started":1}
W20260620 07:00:44.999032 15672 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37161: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
I20260620 07:00:44.999953 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:44.999559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:36941)
0620 07:00:44.999769 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:44.999842 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 07:00:45.000855 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:45.000514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:35839)
0620 07:00:45.000663 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:45.000739 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 07:00:45.001077 15672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:00:45.001888 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15544
I20260620 07:00:45.002527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37079
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:37161
--webserver_port=43275
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:00:45.325762 15675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:00:45.326056 15675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:00:45.326164 15675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:00:45.333446 15675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:00:45.333559 15675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:00:45.333673 15675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:00:45.343755 15675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43275
--tserver_master_addrs=127.0.97.62:37161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:00:45.345014 15675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:00:45.346285 15675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:00:45.359380 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:45.359431 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:00:45.359830 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:00:45.360111 15675 server_base.cc:1034] running on GCE node
I20260620 07:00:45.360742 15675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:00:45.361824 15675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:00:45.363085 15675 hybrid_clock.cc:648] HybridClock initialized: now 1781938845363005 us; error 55 us; skew 500 ppm
I20260620 07:00:45.363466 15675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:00:45 dist-test-slave-mz20 krb5kdc[14223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:00:45.366938 15675 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:00:45.368559 15675 webserver.cc:460] Webserver started at http://127.0.97.3:43275/ using document root <none> and password file <none>
I20260620 07:00:45.369305 15675 fs_manager.cc:359] Metadata directory not provided
I20260620 07:00:45.369438 15675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:00:45.374958 15675 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 07:00:45.377725 15690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:45.379201 15675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:00:45.379359 15675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "cd265dde62314bdb8ebadcbcaa593dc0"
format_stamp: "Formatted at 2026-06-20 07:00:39 on dist-test-slave-mz20"
server_key: "b3168cf00da08004c7a898502809fac6"
server_key_iv: "691186771a08c1bd8ae4a7bf933daeb7"
server_key_version: "encryptionkey@0"
I20260620 07:00:45.380349 15675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:00:45.391738 15675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:00:45.396705 15675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:00:45.396960 15675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:00:45.397938 15675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:00:45.399389 15675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:00:45.399482 15675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:45.399586 15675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:00:45.399652 15675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:00:45.429946 15675 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37079
I20260620 07:00:45.430011 15802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37079 every 8 connection(s)
I20260620 07:00:45.431442 15675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:00:45.434506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15675
I20260620 07:00:45.436447 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:45.434367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:34607)
0620 07:00:45.435207 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:45.435446 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":351,"threads_started":1}
W20260620 07:00:45.437031 15803 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:37161 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:37161: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
I20260620 07:00:45.437819 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:45.437499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:33687)
0620 07:00:45.437651 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:45.437716 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:00:45.438660 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:45.438333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:40041)
0620 07:00:45.438438 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:45.438506 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 07:00:45.438860 15803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:00:45.518360 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:45.517515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:60949)
0620 07:00:45.518024 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:45.518120 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":245,"threads_started":1}
I20260620 07:00:46.002594 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:46.001750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:46415)
0620 07:00:46.002199 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:46.002291 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":214,"threads_started":1}
I20260620 07:00:46.440229 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:46.439435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:39401)
0620 07:00:46.439896 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:46.440005 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":212,"threads_started":1}
I20260620 07:00:46.520097 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:46.519354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:47745)
0620 07:00:46.519829 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:46.519922 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
I20260620 07:00:47.004570 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:47.003647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:52049)
0620 07:00:47.004258 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:47.004370 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260620 07:00:47.442008 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:47.441137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:59331)
0620 07:00:47.441700 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:47.441796 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":283,"threads_started":1}
I20260620 07:00:47.522083 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:47.521108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:35709)
0620 07:00:47.521774 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:47.521886 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
I20260620 07:00:48.006336 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:48.005524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:42353)
0620 07:00:48.006029 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:48.006126 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260620 07:00:48.443770 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:48.442954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:47439)
0620 07:00:48.443473 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:48.443560 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":297,"threads_started":1}
I20260620 07:00:48.523912 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:48.523090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:50631)
0620 07:00:48.523606 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:48.523698 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":260,"threads_started":1}
I20260620 07:00:49.008217 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:49.007359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:45505)
0620 07:00:49.007877 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:49.007975 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":263,"threads_started":1}
I20260620 07:00:49.446000 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:49.444907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:42351)
0620 07:00:49.445539 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:49.445656 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":316,"threads_started":1}
I20260620 07:00:49.525909 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:49.525042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:47989)
0620 07:00:49.525621 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:49.525715 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260620 07:00:50.010438 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:50.009483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:34765)
0620 07:00:50.010128 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:50.010220 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":371,"threads_started":1}
I20260620 07:00:50.448288 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:50.447258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:43881)
0620 07:00:50.447909 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:50.448062 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":336,"threads_started":1}
I20260620 07:00:50.527803 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:50.526960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:38629)
0620 07:00:50.527478 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:50.527576 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":264,"threads_started":1}
I20260620 07:00:51.012457 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:51.011471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:59791)
0620 07:00:51.012075 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:51.012244 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":329,"threads_started":1}
I20260620 07:00:51.450352 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:51.449357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:42935)
0620 07:00:51.449996 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:51.450121 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260620 07:00:51.529948 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:51.528916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:34099)
0620 07:00:51.529575 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:51.529671 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":327,"threads_started":1}
I20260620 07:00:52.014770 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:52.013540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:60181)
0620 07:00:52.014275 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:52.014425 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":298,"threads_started":1}
I20260620 07:00:52.452807 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:52.451739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:44387)
0620 07:00:52.452398 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:52.452522 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":281,"threads_started":1}
I20260620 07:00:52.532106 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:52.531203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:55511)
0620 07:00:52.531757 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:52.531869 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":256,"threads_started":1}
I20260620 07:00:53.016752 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:53.015894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:34793)
0620 07:00:53.016477 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:53.016566 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260620 07:00:53.454625 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:53.453825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:37531)
0620 07:00:53.454315 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:53.454414 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":258,"threads_started":1}
I20260620 07:00:53.533941 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:53.533105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:59359)
0620 07:00:53.533639 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:53.533762 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260620 07:00:54.018607 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:54.017722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:40509)
0620 07:00:54.018270 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:54.018357 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":270,"threads_started":1}
I20260620 07:00:54.456825 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:54.455803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:50243)
0620 07:00:54.456456 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:54.456582 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":308,"threads_started":1}
I20260620 07:00:54.536026 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:54.535089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:35697)
0620 07:00:54.535671 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:54.535775 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
I20260620 07:00:55.020460 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:55.019611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:45877)
0620 07:00:55.020140 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:55.020236 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260620 07:00:55.459043 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:55.458044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:59623)
0620 07:00:55.458685 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:55.458813 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":353,"threads_started":1}
I20260620 07:00:55.538151 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:55.537272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:54313)
0620 07:00:55.537823 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:55.537935 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260620 07:00:56.022323 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:56.021526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:38341)
0620 07:00:56.022008 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:56.022106 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260620 07:00:56.461306 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:56.460285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:42089)
0620 07:00:56.460923 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:56.461050 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":306,"threads_started":1}
I20260620 07:00:56.540120 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:56.539232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:60109)
0620 07:00:56.539790 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:56.539897 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
I20260620 07:00:57.024363 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:57.023421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:44087)
0620 07:00:57.024008 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:57.024151 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":276,"threads_started":1}
I20260620 07:00:57.463446 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:57.462467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:51099)
0620 07:00:57.463117 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:57.463233 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":329,"threads_started":1}
I20260620 07:00:57.542187 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:57.541170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:49735)
0620 07:00:57.541837 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:57.541974 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":311,"threads_started":1}
I20260620 07:00:58.026387 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:58.025477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:40693)
0620 07:00:58.026032 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:58.026128 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":286,"threads_started":1}
I20260620 07:00:58.465529 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:58.464617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:53187)
0620 07:00:58.465187 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:58.465299 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
I20260620 07:00:58.544109 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:58.543237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:50265)
0620 07:00:58.543766 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:58.543861 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260620 07:00:59.028434 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:59.027473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:60479)
0620 07:00:59.028085 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:59.028194 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":314,"threads_started":1}
I20260620 07:00:59.467629 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:59.466646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:38113)
0620 07:00:59.467213 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:59.467325 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260620 07:00:59.546159 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:00:59.545381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.1:58553)
0620 07:00:59.545867 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 07:00:59.545957 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260620 07:01:00.030864 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:00.029865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.2:40673)
0620 07:01:00.030470 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:00.030639 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":296,"threads_started":1}
I20260620 07:01:00.444432 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15379
I20260620 07:01:00.465490 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:00.445854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.97.1:39275)
0620 07:01:00.446749 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:00.446776 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:00.446830 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:01:00.449145 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:00.449161 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:00.449938 (+ 777us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:00.450453 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:00.452110 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.453536 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.454984 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.455812 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.456003 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:00.458709 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:00.458718 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:00.458740 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:00.458786 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:00.461932 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.462677 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.462683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.462691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.462873 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.463227 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.463230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.463233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.463848 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:00.464129 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:00.464597 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 07:01:00.465136 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":411,"threads_started":1}
I20260620 07:01:00.467414 15536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:00.474092 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:00.469027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37161 (local address 127.0.97.3:50353)
0620 07:01:00.473699 (+ 4672us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:00.473850 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:37161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4500,"thread_start_us":4332,"threads_started":1}
I20260620 07:01:00.556782 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15379
I20260620 07:01:00.572003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15544
I20260620 07:01:00.592257 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:00.573191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.0.97.2:40469)
0620 07:01:00.573956 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:00.573975 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:00.574032 (+ 57us) server_negotiation.cc:407] Connection header received
0620 07:01:00.576061 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:00.576091 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:00.576670 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:00.577068 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:00.578680 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.580154 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.581552 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.582322 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.582522 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:00.585145 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:00.585153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:00.585174 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:00.585220 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:00.588645 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.589413 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.589454 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.589462 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.589662 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.589980 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.589983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.589985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.590716 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:00.590978 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:00.591392 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 07:01:00.591907 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":342,"threads_started":1}
I20260620 07:01:00.594197 15667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:00.681070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15544
I20260620 07:01:00.696002 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15675
I20260620 07:01:00.716472 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:00.697254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.0.97.3:37079)
0620 07:01:00.697998 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:00.698016 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:00.698064 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:01:00.699863 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:00.699874 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:00.700422 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:00.700773 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:00.702390 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.703847 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.705339 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:00.706152 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:00.706331 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:00.709462 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:00.709471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:00.709491 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:00.709539 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:00.712691 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.713524 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.713532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.713539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.713736 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:00.714228 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:00.714232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:00.714234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:00.714958 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:00.715216 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:00.715702 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 07:01:00.716115 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":370,"threads_started":1}
I20260620 07:01:00.718116 15798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:00.811769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15675
2026-06-20T07:01:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24746 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:01:00 dist-test-slave-mz20 krb5kdc[15861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:01:00 dist-test-slave-mz20 krb5kdc[15861](info): set up 2 sockets
Jun 20 07:01:00 dist-test-slave-mz20 krb5kdc[15861](info): commencing operation
krb5kdc: starting...
W20260620 07:01:01.863198 388 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:01:02 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:01:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:01:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:01:02.099735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:02.433450 15874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:02.433779 15874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:02.433894 15874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:02.441532 15874 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:01:02.441648 15874 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:01:02.441685 15874 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:01:02.441735 15874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:01:02.441812 15874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:02.441900 15874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:02.441948 15874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:01:02.442018 15874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:01:02.452517 15874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:02.453922 15874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:02.455595 15874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:02.468905 15879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:02.468950 15880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:02.468895 15882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:02.468907 15874 server_base.cc:1034] running on GCE node
I20260620 07:01:02.470233 15874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:02.471801 15874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:02.473029 15874 hybrid_clock.cc:648] HybridClock initialized: now 1781938862472940 us; error 71 us; skew 500 ppm
I20260620 07:01:02.473367 15874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:02 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:02.476581 15874 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:01:02.478103 15874 webserver.cc:460] Webserver started at http://127.0.97.62:38463/ using document root <none> and password file <none>
I20260620 07:01:02.479038 15874 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:02.479154 15874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:02.479530 15874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:01:02.482705 15874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "384a35806303498a8feaa2c985e31b2b"
format_stamp: "Formatted at 2026-06-20 07:01:02 on dist-test-slave-mz20"
server_key: "2813c069374435878517e03e12c78576"
server_key_iv: "bed7526af34666ee608b78df04fe45d2"
server_key_version: "encryptionkey@0"
I20260620 07:01:02.483507 15874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "384a35806303498a8feaa2c985e31b2b"
format_stamp: "Formatted at 2026-06-20 07:01:02 on dist-test-slave-mz20"
server_key: "2813c069374435878517e03e12c78576"
server_key_iv: "bed7526af34666ee608b78df04fe45d2"
server_key_version: "encryptionkey@0"
I20260620 07:01:02.489166 15874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:01:02.492832 15889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:02.494289 15874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:01:02.494477 15874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "384a35806303498a8feaa2c985e31b2b"
format_stamp: "Formatted at 2026-06-20 07:01:02 on dist-test-slave-mz20"
server_key: "2813c069374435878517e03e12c78576"
server_key_iv: "bed7526af34666ee608b78df04fe45d2"
server_key_version: "encryptionkey@0"
I20260620 07:01:02.494731 15874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:02.515419 15874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:02.520061 15874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:02.520332 15874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:02.538555 15874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34755
I20260620 07:01:02.538574 15940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34755 every 8 connection(s)
I20260620 07:01:02.539991 15874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:01:02.542467 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15874
I20260620 07:01:02.542696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:01:02.543394 388 external_mini_cluster.cc:1428] Setting key 0239ea431d6e1fadaf3dca1438edaf5c
I20260620 07:01:02.544360 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:01:02 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:02.559521 15941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap starting.
I20260620 07:01:02.563668 15941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:02.565052 15941 log.cc:824] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:02.568732 15941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: No bootstrap required, opened a new log
I20260620 07:01:02.569339 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:02.545556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.0.97.62:34755)
0620 07:01:02.546415 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:02.546432 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:02.546487 (+ 55us) server_negotiation.cc:407] Connection header received
0620 07:01:02.548227 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:02.548260 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:02.548869 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:02.549412 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:02.551244 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:02.553399 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:02.555070 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:02.556221 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:02.556443 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:02.560590 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:02.560603 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:02.560623 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:02.560681 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:02.564614 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:02.565417 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:02.565426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:02.565435 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:02.565679 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:02.566091 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:02.566094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:02.566096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:02.566784 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:02.567086 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:02.567491 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 07:01:02.567940 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":393,"threads_started":1}
I20260620 07:01:02.576761 15941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:02.577172 15941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:02.577250 15941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Initialized, Role: FOLLOWER
I20260620 07:01:02.577812 15941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:02.578012 15941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:01:02.578112 15941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:01:02.578241 15941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:02.580439 15941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:02.580971 15941 leader_election.cc:302] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384a35806303498a8feaa2c985e31b2b; no voters:
I20260620 07:01:02.581597 15941 leader_election.cc:288] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:01:02.581672 15946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:01:02.582976 15946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 LEADER]: Becoming Leader. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Running, Role: LEADER
I20260620 07:01:02.583607 15946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:02.584192 15941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:01:02.586444 15947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384a35806303498a8feaa2c985e31b2b. Latest consensus state: current_term: 1 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:02.586966 15947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:02.587651 15948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:02.587877 15954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:01:02.588038 15948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:02.593339 15954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:01:02.601922 15954 catalog_manager.cc:1347] Generated new cluster ID: c77d1345885b45168fb30c44e4f1a480
I20260620 07:01:02.602057 15954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:01:02.615108 15954 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:01:02.616441 15954 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:01:02.631219 15954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Generated new TSK 0
I20260620 07:01:02.632463 15954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:01:02.686388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:03.016103 15967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:03.016409 15967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:03.016557 15967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:03.024104 15967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:03.024271 15967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:03.024457 15967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:01:03.034610 15967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:03.036005 15967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:03.037427 15967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:03.051260 15972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:03.051311 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:03.051254 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:03.052098 15967 server_base.cc:1034] running on GCE node
I20260620 07:01:03.052794 15967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:03.053854 15967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:03.055091 15967 hybrid_clock.cc:648] HybridClock initialized: now 1781938863055006 us; error 74 us; skew 500 ppm
I20260620 07:01:03.055441 15967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:03 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:03.058719 15967 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:01:03.060293 15967 webserver.cc:460] Webserver started at http://127.0.97.1:43717/ using document root <none> and password file <none>
I20260620 07:01:03.060992 15967 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:03.061111 15967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:03.061425 15967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:01:03.064625 15967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "dbc9bdcc26924f98927cbaf74cad293f"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "04b5bb8dbe4d8379dbb0005ff026f13d"
server_key_iv: "90bcae4c8c46325cf55ab862d0a302b4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.065382 15967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dbc9bdcc26924f98927cbaf74cad293f"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "04b5bb8dbe4d8379dbb0005ff026f13d"
server_key_iv: "90bcae4c8c46325cf55ab862d0a302b4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.071153 15967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:01:03.074906 15982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.076385 15967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:01:03.076581 15967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "dbc9bdcc26924f98927cbaf74cad293f"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "04b5bb8dbe4d8379dbb0005ff026f13d"
server_key_iv: "90bcae4c8c46325cf55ab862d0a302b4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.076784 15967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:03.101006 15967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:03.105732 15967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:03.106042 15967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:03.107141 15967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:03.108525 15967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:03.108626 15967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.108737 15967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:03.108783 15967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.140017 15967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33015
I20260620 07:01:03.140059 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33015 every 8 connection(s)
I20260620 07:01:03.141269 15967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:01:03.149689 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 15967
I20260620 07:01:03.149896 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:01:03.150733 388 external_mini_cluster.cc:1428] Setting key 2e9f91a79467a953f19a2a75da0cdb17
Jun 20 07:01:03 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:03.162793 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:03.143799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45595 (local address 127.0.97.62:34755)
0620 07:01:03.144272 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:03.144278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:03.145325 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 07:01:03.147263 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:03.147268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:03.147341 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:03.147438 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:03.149034 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.149833 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.152172 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.152771 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.152894 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:03.157013 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:03.157019 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:03.157023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:03.157056 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:03.159617 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:03.160408 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:03.160412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:03.160414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:03.160539 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:03.160979 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:03.160983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:03.160985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:03.161263 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:03.161399 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:03.162156 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 07:01:03.162395 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":327,"thread_start_us":207,"threads_started":1}
I20260620 07:01:03.163852 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:03.144434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:45595)
0620 07:01:03.145159 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:03.145200 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:03.146995 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:03.147682 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:03.147691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:03.148220 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:03.148830 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:03.148849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.150038 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.150045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:03.152001 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:03.152016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.153032 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.153041 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:03.153378 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:03.154117 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:03.154154 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:03.156778 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:03.159768 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:03.159779 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:03.159797 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:03.160269 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:03.160644 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:03.160650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:03.160653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:03.160847 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:03.161385 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:03.161393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:03.161757 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 07:01:03.162022 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 07:01:03.162639 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":282,"threads_started":1}
I20260620 07:01:03.165532 16095 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:03.165901 16095 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:03.166842 16095 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:03.169093 15906 ts_manager.cc:194] Registered new tserver with Master: dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:03.170612 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45595
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:01:03.201470 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:03.531754 16101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:03.532051 16101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:03.532146 16101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:03.539564 16101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:03.539692 16101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:03.539840 16101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:01:03.549894 16101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:03.551231 16101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:03.552577 16101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:03.565055 16106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:03.565127 16107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:03.565263 16109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:03.566205 16101 server_base.cc:1034] running on GCE node
I20260620 07:01:03.566854 16101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:03.567754 16101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:03.568976 16101 hybrid_clock.cc:648] HybridClock initialized: now 1781938863568919 us; error 46 us; skew 500 ppm
I20260620 07:01:03.569288 16101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:03 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:03.572284 16101 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:01:03.573801 16101 webserver.cc:460] Webserver started at http://127.0.97.2:42071/ using document root <none> and password file <none>
I20260620 07:01:03.574587 16101 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:03.574718 16101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:03.575006 16101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:01:03.577859 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "d1f59d78a6154bccabf708bb48244910"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "a11ddec0b958d9496f0def15e4ee2eba"
server_key_iv: "a29a2a17afa6badd8c7b8a8a874fbbb4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.578653 16101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f59d78a6154bccabf708bb48244910"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "a11ddec0b958d9496f0def15e4ee2eba"
server_key_iv: "a29a2a17afa6badd8c7b8a8a874fbbb4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.583884 16101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:01:03.587774 16116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.589289 16101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:01:03.589458 16101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "d1f59d78a6154bccabf708bb48244910"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "a11ddec0b958d9496f0def15e4ee2eba"
server_key_iv: "a29a2a17afa6badd8c7b8a8a874fbbb4"
server_key_version: "encryptionkey@0"
I20260620 07:01:03.589696 16101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:03.608441 16101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:03.612790 16101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:03.613087 16101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:03.614128 16101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:03.615599 16101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:03.615680 16101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.615808 16101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:03.615859 16101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:03.646616 16101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38337
I20260620 07:01:03.646667 16228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38337 every 8 connection(s)
I20260620 07:01:03.647841 16101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:01:03.654223 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16101
I20260620 07:01:03.654426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:01:03.655082 388 external_mini_cluster.cc:1428] Setting key 8b37f4ea9372f3634527c53fcec40490
Jun 20 07:01:03 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:03.672196 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:03.650221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34355 (local address 127.0.97.62:34755)
0620 07:01:03.650383 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:03.650388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:03.651675 (+ 1287us) server_negotiation.cc:407] Connection header received
0620 07:01:03.653551 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:03.653557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:03.653632 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:03.653731 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:03.656036 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.656849 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.659314 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.659836 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.659958 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:03.664575 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:03.664583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:03.664588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:03.664629 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:03.668429 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:03.669444 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:03.669450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:03.669453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:03.669626 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:03.670193 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:03.670198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:03.670201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:03.670609 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:03.670772 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:03.671591 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 07:01:03.671851 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 07:01:03.673197 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:03.650705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:34355)
0620 07:01:03.651482 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:03.651545 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:03.653260 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:03.654080 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:03.654093 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:03.654943 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:03.655816 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:03.655841 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.657020 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.657024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:03.659067 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:03.659091 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:03.660114 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:03.660120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:03.660453 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:03.661246 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:03.661288 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:03.664228 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:03.668699 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:03.668709 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:03.668726 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:03.669277 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:03.669788 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:03.669793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:03.669795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:03.670051 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:03.670773 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:03.670781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:03.671185 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 07:01:03.671450 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 07:01:03.671926 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":303,"threads_started":1}
I20260620 07:01:03.675205 16229 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:03.675575 16229 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:03.676421 16229 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:03.678452 15906 ts_manager.cc:194] Registered new tserver with Master: d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
I20260620 07:01:03.679440 15906 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34355
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:01:03.710188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:04.032330 16234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:04.032640 16234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:04.032737 16234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:04.040145 16234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:04.040278 16234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:04.040393 16234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:01:04.050666 16234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:04.052106 16234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:04.053510 16234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:04.066979 16239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:04.066953 16240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:04.067080 16242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:04.067869 16234 server_base.cc:1034] running on GCE node
I20260620 07:01:04.068480 16234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:04.069420 16234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:04.070653 16234 hybrid_clock.cc:648] HybridClock initialized: now 1781938864070586 us; error 50 us; skew 500 ppm
I20260620 07:01:04.070995 16234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:04 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:04.074059 16234 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:01:04.075510 16234 webserver.cc:460] Webserver started at http://127.0.97.3:41035/ using document root <none> and password file <none>
I20260620 07:01:04.076160 16234 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:04.076282 16234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:04.076584 16234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:01:04.079595 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "d3056e2070f445ad82bcc2e8311a24f5"
format_stamp: "Formatted at 2026-06-20 07:01:04 on dist-test-slave-mz20"
server_key: "05b28480c14dcc3a4e438abd0f90dc17"
server_key_iv: "445bf8e478a6ab3c28c0e683059ebf18"
server_key_version: "encryptionkey@0"
I20260620 07:01:04.080315 16234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d3056e2070f445ad82bcc2e8311a24f5"
format_stamp: "Formatted at 2026-06-20 07:01:04 on dist-test-slave-mz20"
server_key: "05b28480c14dcc3a4e438abd0f90dc17"
server_key_iv: "445bf8e478a6ab3c28c0e683059ebf18"
server_key_version: "encryptionkey@0"
I20260620 07:01:04.085608 16234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:01:04.089089 16249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:04.090567 16234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:01:04.090708 16234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "d3056e2070f445ad82bcc2e8311a24f5"
format_stamp: "Formatted at 2026-06-20 07:01:04 on dist-test-slave-mz20"
server_key: "05b28480c14dcc3a4e438abd0f90dc17"
server_key_iv: "445bf8e478a6ab3c28c0e683059ebf18"
server_key_version: "encryptionkey@0"
I20260620 07:01:04.090886 16234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:04.117189 16234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:04.121421 16234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:04.121728 16234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:04.122740 16234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:04.124086 16234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:04.124161 16234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:04.124262 16234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:04.124306 16234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:04.154636 16234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43617
I20260620 07:01:04.154673 16361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43617 every 8 connection(s)
I20260620 07:01:04.155872 16234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:01:04.162587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16234
I20260620 07:01:04.162781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:01:04.163196 388 external_mini_cluster.cc:1428] Setting key 2f98aeaaeb67e6106469a09725baf63d
Jun 20 07:01:04 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:04.174242 16095 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:04.175839 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.158342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44953 (local address 127.0.97.62:34755)
0620 07:01:04.158561 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.158567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.159711 (+ 1144us) server_negotiation.cc:407] Connection header received
0620 07:01:04.161638 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.161643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.161747 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.161838 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:04.163391 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.164277 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.165917 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.166389 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.166495 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.169711 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:04.169717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:04.169722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:04.169764 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:04.172402 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.173193 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.173198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.173202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.173354 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.173873 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.173877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.173880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.174183 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:04.174333 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.175259 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.175499 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:01:04.176853 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.158858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:44953)
0620 07:01:04.159546 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.159583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.161383 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.162032 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.162040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.162569 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:04.163190 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.163209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.164434 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.164438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.165781 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.165796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.166603 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.166609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.166804 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.167409 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:04.167431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:04.169518 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:04.172561 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:04.172569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:04.172583 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:04.173044 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:04.173479 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:04.173483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:04.173485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:04.173688 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:04.174327 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:04.174335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:04.174818 (+ 483us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.175086 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.175642 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
I20260620 07:01:04.178146 16362 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:04.178458 16362 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:04.179178 16362 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:04.180501 15905 ts_manager.cc:194] Registered new tserver with Master: d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617)
I20260620 07:01:04.181381 15905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44953
I20260620 07:01:04.190892 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:01:04.207779 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.195461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.97.62:34755)
0620 07:01:04.195729 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.195736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.196060 (+ 324us) server_negotiation.cc:407] Connection header received
0620 07:01:04.196216 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.196221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.196286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.196374 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:04.197711 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.198478 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.199911 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.200313 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.200410 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.202671 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:04.202678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:04.202682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:04.202718 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:04.205252 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.205946 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.205951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.205953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.206087 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.206406 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.206410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.206412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.206831 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:04.207017 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.207284 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.207454 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:01:04.212218 15905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:01:04.219400 15905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:01:04.242756 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.233214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.97.3:43617)
0620 07:01:04.234009 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.234018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.234059 (+ 41us) server_negotiation.cc:407] Connection header received
0620 07:01:04.234142 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.234149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.234415 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.234673 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:04.235621 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.237152 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.240338 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.241515 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.241650 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.241895 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.242175 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.242360 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":456,"threads_started":1}
I20260620 07:01:04.242818 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.232874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.0.1:53298)
0620 07:01:04.233571 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.233626 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.233871 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.235028 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.235036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.235066 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:04.235416 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.235430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.237452 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.237458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.240161 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.240176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.241713 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.241720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.241921 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.241975 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.242121 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.242370 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":128,"thread_start_us":362,"threads_started":1}
I20260620 07:01:04.243243 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.234076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:45070)
0620 07:01:04.234611 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.234628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.234743 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.235877 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.235882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.235907 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:04.236370 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.236383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.238319 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.238325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.240623 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.240637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.242390 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.242396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.242611 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.242644 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.242723 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.242861 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":172,"thread_start_us":249,"threads_started":1}
I20260620 07:01:04.243894 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.234448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.97.1:33015)
0620 07:01:04.235275 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.235283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.235312 (+ 29us) server_negotiation.cc:407] Connection header received
0620 07:01:04.235406 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.235414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.235663 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.236583 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:04.236812 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.238189 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.240793 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.242172 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.243079 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.243334 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.243407 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.243575 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":520,"threads_started":1}
I20260620 07:01:04.245450 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.235208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38337 (local address 127.0.0.1:51544)
0620 07:01:04.235943 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.235959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.236156 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.236945 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.236949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.236974 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:04.237355 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.237369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.239264 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.239277 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.241211 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.241226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.243819 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.243826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.244789 (+ 963us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.244824 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.244923 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.245105 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":59,"thread_start_us":462,"threads_started":1}
I20260620 07:01:04.245455 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.234920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.0.97.2:38337)
0620 07:01:04.235523 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.235531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.236051 (+ 520us) server_negotiation.cc:407] Connection header received
0620 07:01:04.236349 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.236358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.236589 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.236755 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:04.237596 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.239082 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.241380 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.243386 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.243505 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.243688 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.244999 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.245182 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":393,"threads_started":1}
I20260620 07:01:04.247565 16297 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa (DEFAULT_TABLE table=test-table [id=c3646839f43c40f1a3021b74f55839b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:04.247609 16030 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa (DEFAULT_TABLE table=test-table [id=c3646839f43c40f1a3021b74f55839b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:04.249068 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c4818c1b64e62ae4685dd50e3abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:04.249290 16030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c4818c1b64e62ae4685dd50e3abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:04.249343 16164 tablet_service.cc:1467] Processing CreateTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa (DEFAULT_TABLE table=test-table [id=c3646839f43c40f1a3021b74f55839b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:04.250789 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f2c4818c1b64e62ae4685dd50e3abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:04.263302 16380 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Bootstrap starting.
I20260620 07:01:04.267127 16381 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Bootstrap starting.
I20260620 07:01:04.267472 16380 tablet_bootstrap.cc:654] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:04.268846 16380 log.cc:824] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:04.270130 16382 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Bootstrap starting.
I20260620 07:01:04.270803 16381 tablet_bootstrap.cc:654] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:04.272060 16381 log.cc:824] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:04.272341 16380 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: No bootstrap required, opened a new log
I20260620 07:01:04.272624 16380 ts_tablet_manager.cc:1397] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 07:01:04.272982 16382 tablet_bootstrap.cc:654] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:04.273983 16382 log.cc:824] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:04.275064 16381 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: No bootstrap required, opened a new log
I20260620 07:01:04.275336 16381 ts_tablet_manager.cc:1397] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260620 07:01:04.276567 16382 tablet_bootstrap.cc:492] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: No bootstrap required, opened a new log
I20260620 07:01:04.276875 16382 ts_tablet_manager.cc:1397] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:01:04.283179 16380 raft_consensus.cc:348] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.283767 16380 raft_consensus.cc:374] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:04.283934 16380 raft_consensus.cc:729] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3056e2070f445ad82bcc2e8311a24f5, State: Initialized, Role: FOLLOWER
I20260620 07:01:04.284345 16382 raft_consensus.cc:348] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.284731 16382 raft_consensus.cc:374] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:04.284827 16382 raft_consensus.cc:729] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc9bdcc26924f98927cbaf74cad293f, State: Initialized, Role: FOLLOWER
I20260620 07:01:04.284724 16380 consensus_queue.cc:260] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.285497 16382 consensus_queue.cc:260] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.285631 16381 raft_consensus.cc:348] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.286017 16362 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:04.286130 16381 raft_consensus.cc:374] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:04.286263 16381 raft_consensus.cc:729] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f59d78a6154bccabf708bb48244910, State: Initialized, Role: FOLLOWER
I20260620 07:01:04.286996 16381 consensus_queue.cc:260] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.288156 16380 ts_tablet_manager.cc:1428] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 07:01:04.292044 16382 ts_tablet_manager.cc:1428] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260620 07:01:04.293190 16229 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:04.294041 16381 ts_tablet_manager.cc:1428] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Time spent starting tablet: real 0.019s user 0.009s sys 0.010s
I20260620 07:01:04.334509 16388 raft_consensus.cc:482] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:04.334909 16388 raft_consensus.cc:504] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.336640 16388 leader_election.cc:288] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:04.344202 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.337140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50599 (local address 127.0.97.3:43617)
0620 07:01:04.337342 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.337348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.337896 (+ 548us) server_negotiation.cc:407] Connection header received
0620 07:01:04.338152 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.338159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.338244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.338303 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:04.339078 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.339970 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.342153 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.343034 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.343138 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.343231 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.343791 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.343939 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:01:04.344220 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.338295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42145 (local address 127.0.97.1:33015)
0620 07:01:04.338521 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.338527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.338572 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:01:04.338786 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.338791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.338867 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.338959 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:04.339570 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.340412 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.342293 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.343282 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.343430 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.343530 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.343860 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.343996 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:01:04.344338 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.337050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.2:50599)
0620 07:01:04.337793 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.337812 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.337947 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.338517 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.338521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.338583 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:04.338925 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.338938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.340117 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.340122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.341958 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.341972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.343323 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.343330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.343594 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.343623 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.343714 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.343849 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":164,"thread_start_us":432,"threads_started":1}
I20260620 07:01:04.344363 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.337911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.97.2:42145)
0620 07:01:04.338490 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:04.338504 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:04.338650 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:04.338977 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:04.338980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:04.339003 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:04.339380 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.339392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.340578 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.340582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.342098 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:04.342113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.343503 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.343508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:04.343691 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.343723 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:01:04.343813 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:01:04.343939 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":49,"thread_start_us":276,"threads_started":1}
I20260620 07:01:04.345409 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2c4818c1b64e62ae4685dd50e3abfa" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3056e2070f445ad82bcc2e8311a24f5" is_pre_election: true
I20260620 07:01:04.345409 16050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2c4818c1b64e62ae4685dd50e3abfa" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc9bdcc26924f98927cbaf74cad293f" is_pre_election: true
I20260620 07:01:04.345927 16050 raft_consensus.cc:2454] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 0.
I20260620 07:01:04.345921 16317 raft_consensus.cc:2454] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 0.
I20260620 07:01:04.346681 16120 leader_election.cc:302] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f59d78a6154bccabf708bb48244910, dbc9bdcc26924f98927cbaf74cad293f; no voters:
I20260620 07:01:04.347158 16388 raft_consensus.cc:2781] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:01:04.347322 16388 raft_consensus.cc:482] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:01:04.347431 16388 raft_consensus.cc:3037] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:04.349786 16388 raft_consensus.cc:504] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.350486 16388 leader_election.cc:288] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 election: Requested vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:04.351078 16050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2c4818c1b64e62ae4685dd50e3abfa" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc9bdcc26924f98927cbaf74cad293f"
I20260620 07:01:04.351069 16317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f2c4818c1b64e62ae4685dd50e3abfa" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3056e2070f445ad82bcc2e8311a24f5"
I20260620 07:01:04.351291 16317 raft_consensus.cc:3037] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:04.351277 16050 raft_consensus.cc:3037] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:04.353574 16050 raft_consensus.cc:2454] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 1.
I20260620 07:01:04.353602 16317 raft_consensus.cc:2454] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 1.
I20260620 07:01:04.354031 16120 leader_election.cc:302] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5; no voters:
I20260620 07:01:04.354346 16388 raft_consensus.cc:2781] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:01:04.354801 16388 raft_consensus.cc:686] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 1 LEADER]: Becoming Leader. State: Replica: d1f59d78a6154bccabf708bb48244910, State: Running, Role: LEADER
I20260620 07:01:04.355362 16388 consensus_queue.cc:237] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:04.359994 15906 catalog_manager.cc:5495] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f59d78a6154bccabf708bb48244910 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d1f59d78a6154bccabf708bb48244910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } health_report { overall_health: HEALTHY } } }
W20260620 07:01:04.393637 16096 tablet.cc:2369] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:01:04.399740 16230 tablet.cc:2369] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:01:04.405164 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.399767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.0.97.2:38337)
0620 07:01:04.399968 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.399974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.400072 (+ 98us) server_negotiation.cc:407] Connection header received
0620 07:01:04.400278 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.400284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.400384 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.400541 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:01:04.400845 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.401554 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.403044 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.403490 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.403588 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.404154 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:01:04.404682 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:01:04.404821 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.404859 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.404933 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 07:01:04.407388 16363 tablet.cc:2369] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:01:04.413776 16050 raft_consensus.cc:1261] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Refusing update from remote peer d1f59d78a6154bccabf708bb48244910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:01:04.413779 16317 raft_consensus.cc:1261] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f59d78a6154bccabf708bb48244910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:01:04.414780 16388 consensus_queue.cc:1035] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:01:04.415139 16391 consensus_queue.cc:1035] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:01:04.440591 15905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:01:04.440871 15905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:01:04.444631 15905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Sending DeleteTablet for 3 replicas of tablet 0f2c4818c1b64e62ae4685dd50e3abfa
I20260620 07:01:04.445822 16297 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53298
I20260620 07:01:04.446002 16030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45070
I20260620 07:01:04.446120 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15967
I20260620 07:01:04.446275 16164 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f2c4818c1b64e62ae4685dd50e3abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51544
I20260620 07:01:04.447211 16406 tablet_replica.cc:330] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: stopping tablet replica
I20260620 07:01:04.447719 16406 raft_consensus.cc:2229] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:01:04.448434 16406 raft_consensus.cc:2258] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:01:04.450798 16406 ts_tablet_manager.cc:1905] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:01:04.459079 16406 ts_tablet_manager.cc:1918] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:01:04.459349 16406 log.cc:1196] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/0f2c4818c1b64e62ae4685dd50e3abfa
I20260620 07:01:04.459933 16406 ts_tablet_manager.cc:1939] T 0f2c4818c1b64e62ae4685dd50e3abfa P d1f59d78a6154bccabf708bb48244910: Deleting consensus metadata
I20260620 07:01:04.461661 16404 tablet_replica.cc:330] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: stopping tablet replica
I20260620 07:01:04.462148 16404 raft_consensus.cc:2229] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:01:04.462735 16404 raft_consensus.cc:2258] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:01:04.463124 15893 catalog_manager.cc:4841] TS d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): tablet 0f2c4818c1b64e62ae4685dd50e3abfa (table test-table [id=c3646839f43c40f1a3021b74f55839b1]) successfully deleted
I20260620 07:01:04.463564 16405 tablet_replica.cc:330] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: stopping tablet replica
I20260620 07:01:04.463948 16405 raft_consensus.cc:2229] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:01:04.464397 16405 raft_consensus.cc:2258] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:01:04.466794 16404 ts_tablet_manager.cc:1905] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:01:04.466794 16405 ts_tablet_manager.cc:1905] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:01:04 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:01:04.473949 16404 ts_tablet_manager.cc:1918] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:01:04.473949 16405 ts_tablet_manager.cc:1918] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:01:04.474126 16405 log.cc:1196] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/0f2c4818c1b64e62ae4685dd50e3abfa
I20260620 07:01:04.474673 16405 ts_tablet_manager.cc:1939] T 0f2c4818c1b64e62ae4685dd50e3abfa P d3056e2070f445ad82bcc2e8311a24f5: Deleting consensus metadata
I20260620 07:01:04.476753 15893 catalog_manager.cc:4841] TS d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): tablet 0f2c4818c1b64e62ae4685dd50e3abfa (table test-table [id=c3646839f43c40f1a3021b74f55839b1]) successfully deleted
I20260620 07:01:04.477758 16404 log.cc:1196] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/0f2c4818c1b64e62ae4685dd50e3abfa
I20260620 07:01:04.478310 16404 ts_tablet_manager.cc:1939] T 0f2c4818c1b64e62ae4685dd50e3abfa P dbc9bdcc26924f98927cbaf74cad293f: Deleting consensus metadata
I20260620 07:01:04.480357 15893 catalog_manager.cc:4841] TS dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015): tablet 0f2c4818c1b64e62ae4685dd50e3abfa (table test-table [id=c3646839f43c40f1a3021b74f55839b1]) successfully deleted
I20260620 07:01:04.483664 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.447560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.0.97.1:33015)
0620 07:01:04.447740 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.447746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.457657 (+ 9911us) server_negotiation.cc:407] Connection header received
0620 07:01:04.457879 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.457886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.458011 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.458113 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:04.460815 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.462472 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.465278 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.465903 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.466000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.471502 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:04.471517 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:04.471538 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:04.471583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:04.480520 (+ 8937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.481464 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.481471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.481474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.481637 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.482090 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.482094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.482096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.482496 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:04.482816 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.483099 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.483332 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:01:04.484300 16090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:04.583281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15967
I20260620 07:01:04.598416 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16101
Jun 20 07:01:04 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:01:04.614300 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.599228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.97.2:38337)
0620 07:01:04.599398 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.599404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.599491 (+ 87us) server_negotiation.cc:407] Connection header received
0620 07:01:04.599675 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.599679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.599746 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.599907 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:04.601153 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.602032 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.603459 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.603911 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.604027 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.608074 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:04.608089 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:04.608105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:04.608152 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:04.611498 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.612218 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.612226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.612228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.612376 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.612750 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.612754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.612759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.613129 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:04.613460 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.613696 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.613926 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:01:04.615146 16224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:04.728370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16101
I20260620 07:01:04.743707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16234
Jun 20 07:01:04 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:01:04.759299 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:04.744445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.0.97.3:43617)
0620 07:01:04.744623 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:04.744630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:04.744694 (+ 64us) server_negotiation.cc:407] Connection header received
0620 07:01:04.744919 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:04.744925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:04.744996 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:04.745077 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:04.746319 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.747091 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.748479 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:04.748876 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:04.748984 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:04.753352 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:04.753370 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:04.753387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:04.753431 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:04.756174 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.757045 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.757054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.757056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.757226 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:04.757642 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:04.757646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:04.757648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:04.758059 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:04.758373 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:04.758744 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 07:01:04.758976 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:01:04.759967 16357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:04.848935 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16234
I20260620 07:01:04.864322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 15874
I20260620 07:01:04.865450 15936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:04.975250 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 15874
I20260620 07:01:24.993291 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--webserver_interface=127.0.97.62
--webserver_port=38463
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:25.380489 16412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:25.380867 16412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:25.380993 16412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:25.389184 16412 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:01:25.389312 16412 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:01:25.389353 16412 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:01:25.389401 16412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:01:25.389482 16412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:25.389593 16412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:25.389657 16412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:01:25.389737 16412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:01:25.401180 16412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:25.402987 16412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:25.404685 16412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:25.431370 16419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:25.431411 16418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:25.431454 16421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:25.431778 16412 server_base.cc:1034] running on GCE node
I20260620 07:01:25.433282 16412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:25.435276 16412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:25.436595 16412 hybrid_clock.cc:648] HybridClock initialized: now 1781938885436461 us; error 88 us; skew 500 ppm
I20260620 07:01:25.437100 16412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:25 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:25.441399 16412 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:01:25.443538 16412 webserver.cc:460] Webserver started at http://127.0.97.62:38463/ using document root <none> and password file <none>
I20260620 07:01:25.444511 16412 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:25.444648 16412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:25.452011 16412 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:01:25.455618 16428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:25.457520 16412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:01:25.457743 16412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "384a35806303498a8feaa2c985e31b2b"
format_stamp: "Formatted at 2026-06-20 07:01:02 on dist-test-slave-mz20"
server_key: "2813c069374435878517e03e12c78576"
server_key_iv: "bed7526af34666ee608b78df04fe45d2"
server_key_version: "encryptionkey@0"
I20260620 07:01:25.459036 16412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:25.492909 16412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:25.497872 16412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:25.498163 16412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:25.518260 16412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34755
I20260620 07:01:25.518337 16479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34755 every 8 connection(s)
I20260620 07:01:25.520219 16412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:01:25.521340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16412
I20260620 07:01:25.522329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33015
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=43717
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:01:25.528311 16480 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:01:25.532081 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap starting.
I20260620 07:01:25.560765 16480 log.cc:824] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:25.576697 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:01:25.577378 16480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap complete.
I20260620 07:01:25.591014 16480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:25.592583 16480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Initialized, Role: FOLLOWER
I20260620 07:01:25.593549 16480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:25.593878 16480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:01:25.594027 16480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:01:25.594233 16480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 1 FOLLOWER]: Advancing to term 2
I20260620 07:01:25.600260 16480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:25.600931 16480 leader_election.cc:302] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384a35806303498a8feaa2c985e31b2b; no voters:
I20260620 07:01:25.601595 16480 leader_election.cc:288] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 07:01:25.601745 16484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 07:01:25.602727 16484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 LEADER]: Becoming Leader. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Running, Role: LEADER
I20260620 07:01:25.603308 16484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:25.603776 16480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:01:25.605903 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:25.605940 16486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384a35806303498a8feaa2c985e31b2b. Latest consensus state: current_term: 2 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:25.606371 16486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:25.606367 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:25.617815 16491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:01:25.626780 16491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:01:25.628424 16491 catalog_manager.cc:1259] Loaded cluster ID: c77d1345885b45168fb30c44e4f1a480
I20260620 07:01:25.628628 16491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:01:25.632900 16491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:01:25.634677 16491 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Loaded TSK: 0
I20260620 07:01:25.684600 16491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Generated new TSK 1
I20260620 07:01:25.684974 16491 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:01:25.946630 16482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:25.946944 16482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:25.947078 16482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:25.954578 16482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:25.954771 16482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:25.954969 16482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:01:25.965121 16482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43717
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:25.966450 16482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:25.967772 16482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:25.982106 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:25.982115 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:25.982478 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:25.984021 16482 server_base.cc:1034] running on GCE node
I20260620 07:01:25.984652 16482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:25.985657 16482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:25.986892 16482 hybrid_clock.cc:648] HybridClock initialized: now 1781938885986827 us; error 57 us; skew 500 ppm
I20260620 07:01:25.987241 16482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:25 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:25.990356 16482 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:01:25.991858 16482 webserver.cc:460] Webserver started at http://127.0.97.1:43717/ using document root <none> and password file <none>
I20260620 07:01:25.992633 16482 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:25.992765 16482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:25.998677 16482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 07:01:26.001611 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.003168 16482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:26.003329 16482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "dbc9bdcc26924f98927cbaf74cad293f"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "04b5bb8dbe4d8379dbb0005ff026f13d"
server_key_iv: "90bcae4c8c46325cf55ab862d0a302b4"
server_key_version: "encryptionkey@0"
I20260620 07:01:26.004364 16482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:26.017120 16482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:26.021543 16482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:26.021786 16482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:26.022797 16482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:26.024168 16482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:26.024262 16482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.024371 16482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:26.024438 16482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.054345 16482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33015
I20260620 07:01:26.054411 16629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33015 every 8 connection(s)
I20260620 07:01:26.055821 16482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:01:26.063663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16482
I20260620 07:01:26.064461 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38337
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=42071
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:01:26 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:26.079648 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.058865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:47775)
0620 07:01:26.059657 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:26.059703 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:26.061397 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:26.062823 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:26.062834 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:26.063625 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:26.064356 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.064382 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.066070 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.066075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.067805 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.067820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.068854 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.068860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.069143 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.069837 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:26.069863 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:26.072299 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:26.075542 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.075551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.075566 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.075966 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.076359 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.076364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.076368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.076549 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.077238 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:26.077246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:26.077558 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 07:01:26.077785 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 07:01:26.078330 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
I20260620 07:01:26.079646 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.058766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47775 (local address 127.0.97.62:34755)
0620 07:01:26.059410 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.059422 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.059818 (+ 396us) server_negotiation.cc:407] Connection header received
0620 07:01:26.061849 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.061862 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.062206 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.062607 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:26.064631 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.065892 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.067966 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.068627 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.068767 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.072627 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:26.072636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:26.072650 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:26.072691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:26.075391 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.076098 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.076104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.076109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.076254 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.076659 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.076663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.076665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.077113 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:26.077293 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:26.078023 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 07:01:26.078392 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":300,"threads_started":1}
I20260620 07:01:26.082260 16630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:26.082553 16630 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:26.083221 16630 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:26.085162 16445 ts_manager.cc:194] Registered new tserver with Master: dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:26.086607 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47775
W20260620 07:01:26.381803 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:26.382086 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:26.382206 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:26.389379 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:26.389498 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:26.389663 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:01:26.399348 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42071
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:26.400590 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:26.401832 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:26.414436 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:26.414368 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:26.414410 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:26.415352 16634 server_base.cc:1034] running on GCE node
I20260620 07:01:26.415920 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:26.416867 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:26.418064 16634 hybrid_clock.cc:648] HybridClock initialized: now 1781938886418008 us; error 51 us; skew 500 ppm
I20260620 07:01:26.418373 16634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:26 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:26.424489 16634 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:01:26.426065 16634 webserver.cc:460] Webserver started at http://127.0.97.2:42071/ using document root <none> and password file <none>
I20260620 07:01:26.426846 16634 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:26.426980 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:26.432247 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:01:26.435004 16649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.436427 16634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:01:26.436635 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "d1f59d78a6154bccabf708bb48244910"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "a11ddec0b958d9496f0def15e4ee2eba"
server_key_iv: "a29a2a17afa6badd8c7b8a8a874fbbb4"
server_key_version: "encryptionkey@0"
I20260620 07:01:26.437677 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:26.451730 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:26.455915 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:26.456175 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:26.457175 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:26.458472 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:26.458623 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.458729 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:26.458798 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.489516 16634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38337
I20260620 07:01:26.489615 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38337 every 8 connection(s)
I20260620 07:01:26.490939 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:01:26.498190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16634
I20260620 07:01:26.498993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43617
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=41035
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 07:01:26 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:26.511298 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.493416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39787 (local address 127.0.97.62:34755)
0620 07:01:26.493623 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.493630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.494811 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 07:01:26.496574 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.496579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.496649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.496739 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:26.498469 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.499548 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.501085 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.501632 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.501751 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.505173 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:26.505180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:26.505185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:26.505274 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:26.508212 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.509028 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.509034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.509037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.509173 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.509601 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.509606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.509609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.509924 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:26.510069 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:26.510758 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 07:01:26.510984 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:01:26.512194 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.493854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:39787)
0620 07:01:26.494657 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:26.494705 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:26.496306 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:26.496989 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:26.496998 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:26.497692 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:26.498238 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.498258 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.499713 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.499717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.500942 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.500954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.501844 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.501849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.502058 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.502769 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:26.502801 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:26.504925 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:26.508347 (+ 3422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.508353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.508366 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.508848 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.509307 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.509311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.509314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.509500 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.510076 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:26.510082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:26.510400 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 07:01:26.510667 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 07:01:26.511064 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":291,"threads_started":1}
I20260620 07:01:26.513582 16762 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:26.513895 16762 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:26.514730 16762 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:26.516368 16445 ts_manager.cc:194] Registered new tserver with Master: d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
I20260620 07:01:26.517439 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39787
W20260620 07:01:26.829401 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:26.829674 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:26.829792 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:26.837138 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:26.837255 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:26.837396 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:01:26.847678 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41035
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:26.848981 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:26.850306 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:26.863346 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:26.863423 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:26.863397 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:26.864498 16765 server_base.cc:1034] running on GCE node
I20260620 07:01:26.865132 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:26.866091 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:26.867287 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781938886867234 us; error 44 us; skew 500 ppm
I20260620 07:01:26.867589 16765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:26 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:26.870800 16765 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:01:26.872370 16765 webserver.cc:460] Webserver started at http://127.0.97.3:41035/ using document root <none> and password file <none>
I20260620 07:01:26.873049 16765 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:26.873178 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:26.878562 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:01:26.881295 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.882702 16765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:26.882862 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "d3056e2070f445ad82bcc2e8311a24f5"
format_stamp: "Formatted at 2026-06-20 07:01:04 on dist-test-slave-mz20"
server_key: "05b28480c14dcc3a4e438abd0f90dc17"
server_key_iv: "445bf8e478a6ab3c28c0e683059ebf18"
server_key_version: "encryptionkey@0"
I20260620 07:01:26.883852 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:26.896602 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:26.900560 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:26.900810 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:26.901738 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:26.903059 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:26.903156 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.903256 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:26.903326 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:26.933123 16765 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43617
I20260620 07:01:26.933183 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43617 every 8 connection(s)
I20260620 07:01:26.934486 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:01:26.935395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16765
Jun 20 07:01:26 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:01:26.951458 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.936429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.0.97.62:34755)
0620 07:01:26.936632 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.936639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.937062 (+ 423us) server_negotiation.cc:407] Connection header received
0620 07:01:26.937245 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.937250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.937328 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.937457 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:26.939538 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.940300 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.942089 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.942604 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.942721 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.945407 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:26.945412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:26.945416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:26.945447 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:26.948648 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.949359 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.949365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.949368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.949512 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.950004 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.950008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.950010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.950330 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:26.950565 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:26.950948 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 07:01:26.951120 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":63}
I20260620 07:01:26.956147 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.937038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50261 (local address 127.0.97.62:34755)
0620 07:01:26.937873 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.937877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.938703 (+ 826us) server_negotiation.cc:407] Connection header received
0620 07:01:26.940781 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.940785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.940854 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.940927 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:26.942669 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.943683 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.945201 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.945742 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.945829 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.950021 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:26.950028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:26.950031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:26.950076 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:26.953027 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.953805 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.953809 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.953812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.953943 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.954417 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.954422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.954424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.954768 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:26.954898 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:26.955575 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 07:01:26.955777 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":729,"thread_start_us":654,"threads_started":1}
I20260620 07:01:26.957008 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.937678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:50261)
0620 07:01:26.938522 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:26.938598 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:26.940508 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:26.941190 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:26.941202 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:26.941892 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:26.942496 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.942513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.943877 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.943881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.945064 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:26.945076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.945976 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.945984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:26.946257 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.947078 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:26.947115 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:26.949776 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:26.953174 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.953184 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.953201 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.953675 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.954100 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:26.954105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:26.954110 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:26.954299 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:26.954895 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:26.954903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:26.955221 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 07:01:26.955455 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 07:01:26.955956 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":313,"threads_started":1}
I20260620 07:01:26.958192 16893 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34755
I20260620 07:01:26.958441 16893 heartbeater.cc:464] Registering TS with master...
I20260620 07:01:26.959127 16893 heartbeater.cc:511] Master 127.0.97.62:34755 requested a full tablet report, sending...
I20260620 07:01:26.960443 16445 ts_manager.cc:194] Registered new tserver with Master: d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617)
I20260620 07:01:26.961251 16445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50261
I20260620 07:01:26.966367 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:01:26.972770 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.967373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.97.62:34755)
0620 07:01:26.967562 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.967569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.967975 (+ 406us) server_negotiation.cc:407] Connection header received
0620 07:01:26.968148 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.968152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.968289 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.968409 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:01:26.968815 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.969554 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.971121 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.971536 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.971630 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.972101 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:01:26.972310 (+ 209us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 07:01:26.972573 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41586: token expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 07:01:26.972859 16896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41586: token expired
W20260620 07:01:26.973430 16365 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:34755)
I20260620 07:01:26.985965 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:26.973866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.97.62:34755)
0620 07:01:26.974019 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:26.974023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:26.974043 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:01:26.974080 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:26.974083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:26.974137 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:26.974230 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:26.975601 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.976308 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.977700 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:26.978127 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:26.978221 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:26.981099 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:26.981106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:26.981111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:26.981151 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:26.983621 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.984275 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.984279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.984281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.984391 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:26.984786 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:26.984790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:26.984793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:26.985124 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:26.985337 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:26.985564 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:01:26.985732 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:01:26.989472 16445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:01:26.997211 16445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:01:27.018738 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.009140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.0.97.3:43617)
0620 07:01:27.010887 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.010896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.010927 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:01:27.011021 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.011028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.011247 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.011413 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:27.012057 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.013751 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.016238 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.017394 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.017860 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.018157 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.018230 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.018382 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1602,"thread_start_us":1450,"threads_started":1}
I20260620 07:01:27.018798 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.008954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.0.1:35496)
0620 07:01:27.009742 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.009782 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:27.009941 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:27.011441 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:27.011448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:27.011477 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:27.011815 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.011825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.013913 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.013918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.016041 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.016057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.017652 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.017659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.017955 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.018030 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 07:01:27.018162 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 07:01:27.018359 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":45,"thread_start_us":395,"threads_started":1}
I20260620 07:01:27.022634 16828 tablet_service.cc:1467] Processing CreateTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c (DEFAULT_TABLE table=test-table [id=727b58dafd4d4d59b1019f65a41e54d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:27.023418 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.011288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38337 (local address 127.0.0.1:50414)
0620 07:01:27.012439 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.012456 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:27.012595 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:27.013373 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:27.013378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:27.013403 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:27.013755 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.013765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.015594 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.015600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.017946 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.017963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.019974 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.019980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.022725 (+ 2745us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.022762 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 07:01:27.022864 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 07:01:27.023029 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":2,"thread_start_us":908,"threads_started":1}
I20260620 07:01:27.024173 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.011660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.97.2:38337)
0620 07:01:27.012240 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.012248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.012552 (+ 304us) server_negotiation.cc:407] Connection header received
0620 07:01:27.012731 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.012739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.012956 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.013170 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:27.013913 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.015441 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.018148 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.019768 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.019909 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.020102 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.023684 (+ 3582us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.023868 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":257,"threads_started":1}
I20260620 07:01:27.024642 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49535b19561d49a49a1ef0e9f2a88f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:27.027882 16697 tablet_service.cc:1467] Processing CreateTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c (DEFAULT_TABLE table=test-table [id=727b58dafd4d4d59b1019f65a41e54d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:27.029188 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.010104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51740)
0620 07:01:27.010618 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.010634 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:27.010738 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:27.022497 (+ 11759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:27.022502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:27.022559 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:27.022966 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.022980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.025640 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.025646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.027141 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.027155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.028439 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.028446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.028609 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.028636 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 07:01:27.028708 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 07:01:27.028844 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":881,"threads_started":1}
I20260620 07:01:27.029249 16697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49535b19561d49a49a1ef0e9f2a88f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:27.029304 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.010601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.0.97.1:33015)
0620 07:01:27.022032 (+ 11431us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.022039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.022061 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:01:27.022138 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.022143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.022314 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.022480 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:27.023408 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.025502 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.027295 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.028231 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.028347 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.028539 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.028793 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.028951 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11262,"thread_start_us":12084,"threads_started":1}
I20260620 07:01:27.032528 16565 tablet_service.cc:1467] Processing CreateTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c (DEFAULT_TABLE table=test-table [id=727b58dafd4d4d59b1019f65a41e54d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:01:27.033957 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49535b19561d49a49a1ef0e9f2a88f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:01:27.040673 16909 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Bootstrap starting.
I20260620 07:01:27.044461 16910 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Bootstrap starting.
I20260620 07:01:27.044530 16909 tablet_bootstrap.cc:654] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:27.045854 16909 log.cc:824] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:27.047816 16910 tablet_bootstrap.cc:654] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:27.048800 16912 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Bootstrap starting.
I20260620 07:01:27.049036 16910 log.cc:824] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:27.049407 16909 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: No bootstrap required, opened a new log
I20260620 07:01:27.049736 16909 ts_tablet_manager.cc:1397] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 07:01:27.051708 16912 tablet_bootstrap.cc:654] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Neither blocks nor log segments found. Creating new log.
I20260620 07:01:27.052906 16912 log.cc:824] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:27.054682 16910 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: No bootstrap required, opened a new log
I20260620 07:01:27.054965 16910 ts_tablet_manager.cc:1397] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 07:01:27.060887 16909 raft_consensus.cc:348] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.061352 16909 raft_consensus.cc:374] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:27.061527 16909 raft_consensus.cc:729] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3056e2070f445ad82bcc2e8311a24f5, State: Initialized, Role: FOLLOWER
I20260620 07:01:27.062361 16909 consensus_queue.cc:260] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.063342 16912 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: No bootstrap required, opened a new log
I20260620 07:01:27.063725 16912 ts_tablet_manager.cc:1397] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260620 07:01:27.064288 16893 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:27.064882 16909 ts_tablet_manager.cc:1428] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260620 07:01:27.066824 16910 raft_consensus.cc:348] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.067185 16910 raft_consensus.cc:374] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:27.067281 16910 raft_consensus.cc:729] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f59d78a6154bccabf708bb48244910, State: Initialized, Role: FOLLOWER
I20260620 07:01:27.067898 16910 consensus_queue.cc:260] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.069162 16762 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:27.070577 16910 ts_tablet_manager.cc:1428] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Time spent starting tablet: real 0.015s user 0.011s sys 0.001s
I20260620 07:01:27.072881 16912 raft_consensus.cc:348] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.073331 16912 raft_consensus.cc:374] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:01:27.073448 16912 raft_consensus.cc:729] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc9bdcc26924f98927cbaf74cad293f, State: Initialized, Role: FOLLOWER
I20260620 07:01:27.073961 16912 consensus_queue.cc:260] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.074935 16630 heartbeater.cc:503] Master 127.0.97.62:34755 was elected leader, sending a full tablet report...
I20260620 07:01:27.075281 16912 ts_tablet_manager.cc:1428] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260620 07:01:27.186184 16894 tablet.cc:2369] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:01:27.229547 16916 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:27.229835 16916 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.231489 16916 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:27.238458 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.232006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35067 (local address 127.0.97.3:43617)
0620 07:01:27.232235 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.232243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.232632 (+ 389us) server_negotiation.cc:407] Connection header received
0620 07:01:27.232875 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.232881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.232961 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.233022 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:27.233782 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.234669 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.236615 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.237473 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.237570 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.237654 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.238064 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.238202 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:01:27.238482 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.232775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40093 (local address 127.0.97.1:33015)
0620 07:01:27.232950 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.232955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.233254 (+ 299us) server_negotiation.cc:407] Connection header received
0620 07:01:27.233434 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.233439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.233512 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.233616 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:01:27.234200 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.235040 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.236816 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.237551 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.237692 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.237781 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.238131 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.238255 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:01:27.238611 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.231913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.2:35067)
0620 07:01:27.232503 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.232520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:27.232692 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:27.233219 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:27.233224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:27.233251 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:27.233631 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.233645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.234814 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.234819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.236449 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.236461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.237703 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.237709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.237904 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.237929 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:01:27.237998 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:01:27.238115 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":80,"thread_start_us":340,"threads_started":1}
I20260620 07:01:27.238656 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.232600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.97.2:40093)
0620 07:01:27.233153 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.233170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:27.233314 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:27.233644 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:27.233647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:27.233667 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:01:27.234030 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.234041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.235205 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.235209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.236669 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:27.236684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.237778 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.237782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:27.237946 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.237969 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:01:27.238082 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 07:01:27.238210 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":18,"spinlock_wait_cycles":17152,"thread_start_us":260,"threads_started":1}
I20260620 07:01:27.239645 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49535b19561d49a49a1ef0e9f2a88f3c" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3056e2070f445ad82bcc2e8311a24f5" is_pre_election: true
I20260620 07:01:27.239639 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49535b19561d49a49a1ef0e9f2a88f3c" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc9bdcc26924f98927cbaf74cad293f" is_pre_election: true
I20260620 07:01:27.240095 16585 raft_consensus.cc:2454] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 0.
I20260620 07:01:27.240114 16848 raft_consensus.cc:2454] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 0.
I20260620 07:01:27.240728 16653 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f59d78a6154bccabf708bb48244910, dbc9bdcc26924f98927cbaf74cad293f; no voters:
I20260620 07:01:27.241087 16916 raft_consensus.cc:2781] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:01:27.241254 16916 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:01:27.241351 16916 raft_consensus.cc:3037] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 0 FOLLOWER]: Advancing to term 1
W20260620 07:01:27.242825 16763 tablet.cc:2369] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:01:27.244055 16916 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.244604 16916 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 election: Requested vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015)
I20260620 07:01:27.245177 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49535b19561d49a49a1ef0e9f2a88f3c" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3056e2070f445ad82bcc2e8311a24f5"
I20260620 07:01:27.245272 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49535b19561d49a49a1ef0e9f2a88f3c" candidate_uuid: "d1f59d78a6154bccabf708bb48244910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc9bdcc26924f98927cbaf74cad293f"
I20260620 07:01:27.245407 16848 raft_consensus.cc:3037] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:27.245468 16585 raft_consensus.cc:3037] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:01:27.248544 16585 raft_consensus.cc:2454] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 1.
I20260620 07:01:27.248544 16848 raft_consensus.cc:2454] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f59d78a6154bccabf708bb48244910 in term 1.
I20260620 07:01:27.248991 16653 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1f59d78a6154bccabf708bb48244910, dbc9bdcc26924f98927cbaf74cad293f; no voters:
I20260620 07:01:27.249306 16916 raft_consensus.cc:2781] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:01:27.249724 16916 raft_consensus.cc:686] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 1 LEADER]: Becoming Leader. State: Replica: d1f59d78a6154bccabf708bb48244910, State: Running, Role: LEADER
I20260620 07:01:27.250250 16916 consensus_queue.cc:237] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:27.255013 16445 catalog_manager.cc:5495] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f59d78a6154bccabf708bb48244910 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d1f59d78a6154bccabf708bb48244910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } health_report { overall_health: HEALTHY } } }
W20260620 07:01:27.308172 16631 tablet.cc:2369] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:01:27.323611 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.318439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.97.2:38337)
0620 07:01:27.318654 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.318659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.318676 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:01:27.318766 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.318769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.318852 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.318997 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:01:27.319386 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.320103 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.321527 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.321940 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.322044 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.322529 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:01:27.323089 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:01:27.323194 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.323272 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.323341 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:01:27.331655 16848 raft_consensus.cc:1261] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Refusing update from remote peer d1f59d78a6154bccabf708bb48244910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:01:27.331655 16585 raft_consensus.cc:1261] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Refusing update from remote peer d1f59d78a6154bccabf708bb48244910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:01:27.332581 16920 consensus_queue.cc:1035] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:01:27.332904 16916 consensus_queue.cc:1035] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:01:27.348052 16924 mvcc.cc:205] Tried to move back new op lower bound from 7298821682500235264 to 7298821682180816896. Current Snapshot: MvccSnapshot[applied={T|T < 7298821682500235264}]
I20260620 07:01:27.357817 16445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:01:27.358048 16445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:01:27.361456 16445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Sending DeleteTablet for 3 replicas of tablet 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:27.362964 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16482
I20260620 07:01:27.362917 16565 tablet_service.cc:1514] Processing DeleteTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51740
I20260620 07:01:27.362922 16697 tablet_service.cc:1514] Processing DeleteTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50414
I20260620 07:01:27.363660 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet 49535b19561d49a49a1ef0e9f2a88f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:01:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35496
I20260620 07:01:27.389144 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.364220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.0.97.1:33015)
0620 07:01:27.364400 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.364406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.364425 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:01:27.364484 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.364489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.364606 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.364698 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:27.366668 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.367972 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.370042 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.370834 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.370962 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.378473 (+ 7511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:27.378497 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:27.378523 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:27.378604 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:27.382998 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.384108 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.384119 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.384123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.384331 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.385649 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.385654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.385657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.386201 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:27.386494 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.388451 (+ 1957us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.388697 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:01:27.391120 16625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:27.396698 16935 tablet_replica.cc:330] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: stopping tablet replica
I20260620 07:01:27.397836 16935 raft_consensus.cc:2229] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:01:27.399047 16935 raft_consensus.cc:2258] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:01:27.402338 16935 ts_tablet_manager.cc:1905] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:01:27.404141 16934 tablet_replica.cc:330] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: stopping tablet replica
I20260620 07:01:27.408450 16934 raft_consensus.cc:2229] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:01:27.409200 16934 raft_consensus.cc:2258] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:01:27.411641 16934 ts_tablet_manager.cc:1905] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:01:27.413890 16935 ts_tablet_manager.cc:1918] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:01:27.414068 16935 log.cc:1196] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:27.414623 16935 ts_tablet_manager.cc:1939] T 49535b19561d49a49a1ef0e9f2a88f3c P d3056e2070f445ad82bcc2e8311a24f5: Deleting consensus metadata
I20260620 07:01:27.416744 16432 catalog_manager.cc:4841] TS d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): tablet 49535b19561d49a49a1ef0e9f2a88f3c (table test-table [id=727b58dafd4d4d59b1019f65a41e54d9]) successfully deleted
I20260620 07:01:27.420174 16934 ts_tablet_manager.cc:1918] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:01:27.420383 16934 log.cc:1196] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:27.420984 16934 ts_tablet_manager.cc:1939] T 49535b19561d49a49a1ef0e9f2a88f3c P d1f59d78a6154bccabf708bb48244910: Deleting consensus metadata
I20260620 07:01:27.424465 16432 catalog_manager.cc:4841] TS d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): tablet 49535b19561d49a49a1ef0e9f2a88f3c (table test-table [id=727b58dafd4d4d59b1019f65a41e54d9]) successfully deleted
I20260620 07:01:27.515169 16933 tablet_replica.cc:330] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: stopping tablet replica
I20260620 07:01:27.515434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16482
I20260620 07:01:27.515637 16933 raft_consensus.cc:2229] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:01:27.536638 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16634
W20260620 07:01:27.536669 16432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33015 (error 108)
I20260620 07:01:27.546871 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.546165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51760)
0620 07:01:27.546615 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.546722 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":228,"threads_started":1}
W20260620 07:01:27.547452 16432 catalog_manager.cc:4584] TS dbc9bdcc26924f98927cbaf74cad293f (127.0.97.1:33015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 49535b19561d49a49a1ef0e9f2a88f3c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
I20260620 07:01:27.554586 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.538046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.97.2:38337)
0620 07:01:27.538239 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.538245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.538264 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:01:27.538325 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.538330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.538407 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.538504 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:27.540366 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.541660 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.543619 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.544283 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.544404 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.548042 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:27.548062 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:27.548081 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:27.548130 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:27.551629 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.552512 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.552521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.552523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.552719 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.553174 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.553178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.553180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.553567 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:27.553855 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.554098 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.554272 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:01:27.555399 16757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:27.601444 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.600914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51770)
0620 07:01:27.601177 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.601289 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 07:01:27.602730 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.602264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51786)
0620 07:01:27.602436 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.602508 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:01:27.663275 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.662788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51802)
0620 07:01:27.662969 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.663077 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 07:01:27.664337 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.663973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51810)
0620 07:01:27.664079 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.664175 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 07:01:27.670514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16634
I20260620 07:01:27.686838 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16765
I20260620 07:01:27.700613 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.687566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.97.3:43617)
0620 07:01:27.687746 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:27.687751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:27.687829 (+ 78us) server_negotiation.cc:407] Connection header received
0620 07:01:27.687988 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:27.687992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:27.688068 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:27.688150 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:27.689644 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.690405 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.691831 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:27.692268 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:27.692383 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:27.695011 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:27.695028 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:27.695045 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:27.695083 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:27.697580 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.698199 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.698206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.698208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.698346 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:27.699057 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:27.699061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:27.699063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:27.699464 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:27.699695 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:27.700115 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 07:01:27.700290 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:01:27.701277 16888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:27.744910 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.744399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51816)
0620 07:01:27.744636 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.744738 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:01:27.746990 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:27.746464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33015 (local address 127.0.0.1:51830)
0620 07:01:27.746747 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:27.746833 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 07:01:27.790512 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16765
I20260620 07:01:27.804391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 16412
I20260620 07:01:27.805598 16475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:01:27.918110 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 16412
I20260620 07:01:47.936709 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--webserver_interface=127.0.97.62
--webserver_port=38463
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:48.292950 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:48.293262 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:48.293394 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:48.300843 16942 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 07:01:48.300944 16942 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 07:01:48.300985 16942 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 07:01:48.301064 16942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:01:48.301134 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:48.301219 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:48.301280 16942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:01:48.301354 16942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:01:48.311388 16942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:48.312768 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:48.314160 16942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:48.326678 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:48.326699 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:48.326720 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:48.327347 16942 server_base.cc:1034] running on GCE node
I20260620 07:01:48.328142 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:48.329564 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:48.330776 16942 hybrid_clock.cc:648] HybridClock initialized: now 1781938908330701 us; error 52 us; skew 500 ppm
I20260620 07:01:48.331079 16942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:48 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:48.337491 16942 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:01:48.339082 16942 webserver.cc:460] Webserver started at http://127.0.97.62:38463/ using document root <none> and password file <none>
I20260620 07:01:48.339847 16942 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:48.339957 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:48.345817 16942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 07:01:48.348887 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:48.350297 16942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:01:48.350458 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "384a35806303498a8feaa2c985e31b2b"
format_stamp: "Formatted at 2026-06-20 07:01:02 on dist-test-slave-mz20"
server_key: "2813c069374435878517e03e12c78576"
server_key_iv: "bed7526af34666ee608b78df04fe45d2"
server_key_version: "encryptionkey@0"
I20260620 07:01:48.351473 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:48.375414 16942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:48.379927 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:48.380209 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:48.397897 16942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34755
I20260620 07:01:48.397948 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34755 every 8 connection(s)
I20260620 07:01:48.399461 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:01:48.402837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 16942
I20260620 07:01:48.403623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33015
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=43717
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:01:48.406824 17009 sys_catalog.cc:263] Verifying existing consensus state
I20260620 07:01:48.410295 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap starting.
I20260620 07:01:48.438570 17009 log.cc:824] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:48.459137 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:01:48.459877 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Bootstrap complete.
I20260620 07:01:48.473613 17009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:48.474889 17009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Initialized, Role: FOLLOWER
I20260620 07:01:48.475627 17009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:48.475888 17009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:01:48.476013 17009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:01:48.476184 17009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 2 FOLLOWER]: Advancing to term 3
I20260620 07:01:48.481559 17009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:48.482123 17009 leader_election.cc:302] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384a35806303498a8feaa2c985e31b2b; no voters:
I20260620 07:01:48.482818 17009 leader_election.cc:288] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 07:01:48.482962 17013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 07:01:48.484009 17013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [term 3 LEADER]: Becoming Leader. State: Replica: 384a35806303498a8feaa2c985e31b2b, State: Running, Role: LEADER
I20260620 07:01:48.484547 17013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } }
I20260620 07:01:48.484840 17009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:01:48.487022 17015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384a35806303498a8feaa2c985e31b2b. Latest consensus state: current_term: 3 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:48.487097 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "384a35806303498a8feaa2c985e31b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384a35806303498a8feaa2c985e31b2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34755 } } }
I20260620 07:01:48.487469 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:48.487474 17015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b [sys.catalog]: This master's current role is: LEADER
I20260620 07:01:48.498857 17019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:01:48.507243 17019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:01:48.508783 17019 catalog_manager.cc:1259] Loaded cluster ID: c77d1345885b45168fb30c44e4f1a480
I20260620 07:01:48.508946 17019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:01:48.513324 17019 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:01:48.514938 17019 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Loaded TSK: 0
I20260620 07:01:48.515168 17019 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 384a35806303498a8feaa2c985e31b2b: Loaded TSK: 1
F20260620 07:01:48.516530 17019 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcffa2999ad google::LogMessage::Fail() at ??:0
@ 0x7fcffa29c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcffa2994de google::LogMessage::Flush() at ??:0
@ 0x7fcffa29aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcffdffd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcffdfec73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd00fcc81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd00fcc7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd00fd532a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fd00fd530c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fd00fd9d5e9 std::function<>::operator()() at ??:0
@ 0x7fd00fccbcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd00fcc9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd00fd51610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd00fd51439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcffcee5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcffcee891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcffcee8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcffcec3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd00019d6db start_thread at ??:0
@ 0x7fcff913361f clone at ??:0
W20260620 07:01:48.796918 17011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:48.797216 17011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:48.797353 17011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:48.804613 17011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:48.804737 17011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:48.804886 17011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:01:48.814723 17011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43717
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:48.816017 17011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:48.817327 17011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:48.830159 17056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:48.830214 17057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:48.830183 17059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:48.830994 17011 server_base.cc:1034] running on GCE node
I20260620 07:01:48.831602 17011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:48.832525 17011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:48.833763 17011 hybrid_clock.cc:648] HybridClock initialized: now 1781938908833706 us; error 45 us; skew 500 ppm
I20260620 07:01:48.834072 17011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:48 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:48.837090 17011 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:01:48.838590 17011 webserver.cc:460] Webserver started at http://127.0.97.1:43717/ using document root <none> and password file <none>
I20260620 07:01:48.839264 17011 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:48.839376 17011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:48.844595 17011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 07:01:48.847373 17066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:48.848673 17011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:48.848873 17011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "dbc9bdcc26924f98927cbaf74cad293f"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "04b5bb8dbe4d8379dbb0005ff026f13d"
server_key_iv: "90bcae4c8c46325cf55ab862d0a302b4"
server_key_version: "encryptionkey@0"
I20260620 07:01:48.849835 17011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:48.875787 17011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:48.879904 17011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:48.880143 17011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:48.881124 17011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:48.882985 17073 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 07:01:48.888896 17011 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 07:01:48.889034 17011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 07:01:48.889137 17011 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 07:01:48.892398 17011 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 07:01:48.892494 17011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260620 07:01:48.892900 17073 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Bootstrap starting.
I20260620 07:01:48.919145 17073 log.cc:824] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:01:48.927405 17011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33015
I20260620 07:01:48.927407 17180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33015 every 8 connection(s)
I20260620 07:01:48.928951 17011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:01:48.930707 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17011
I20260620 07:01:48.931394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38337
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=42071
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:01:48.933797 17073 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 07:01:48.934275 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:48.932022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:39219)
0620 07:01:48.932935 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:48.933171 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":395,"threads_started":1}
I20260620 07:01:48.934298 17073 tablet_bootstrap.cc:492] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Bootstrap complete.
W20260620 07:01:48.935105 17181 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34755: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
I20260620 07:01:48.935703 17073 ts_tablet_manager.cc:1397] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Time spent bootstrapping tablet: real 0.043s user 0.021s sys 0.020s
I20260620 07:01:48.936030 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:48.935628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:52425)
0620 07:01:48.935794 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:48.935886 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 07:01:48.937003 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:48.936631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:35585)
0620 07:01:48.936767 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:48.936862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 07:01:48.937296 17181 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:01:48.942301 17073 raft_consensus.cc:348] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:48.943246 17073 raft_consensus.cc:729] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc9bdcc26924f98927cbaf74cad293f, State: Initialized, Role: FOLLOWER
I20260620 07:01:48.943814 17073 consensus_queue.cc:260] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:48.945307 17073 ts_tablet_manager.cc:1428] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20260620 07:01:48.945528 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:48.945090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:33543)
0620 07:01:48.945279 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:48.945375 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:01:48.946447 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:48.946083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:45023)
0620 07:01:48.946220 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:48.946295 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 07:01:49.256086 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:49.256369 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:49.256502 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:49.263913 17184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:49.264042 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:49.264165 17184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:01:49.273941 17184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42071
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:49.275287 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:49.276583 17184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:49.289229 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:49.289161 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:49.289160 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:49.290148 17184 server_base.cc:1034] running on GCE node
I20260620 07:01:49.290771 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:49.291682 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:49.292878 17184 hybrid_clock.cc:648] HybridClock initialized: now 1781938909292832 us; error 40 us; skew 500 ppm
I20260620 07:01:49.293180 17184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:49 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:49.296164 17184 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:01:49.297498 17184 webserver.cc:460] Webserver started at http://127.0.97.2:42071/ using document root <none> and password file <none>
I20260620 07:01:49.298137 17184 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:49.298249 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:49.303511 17184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:49.306185 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.307585 17184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:01:49.307736 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "d1f59d78a6154bccabf708bb48244910"
format_stamp: "Formatted at 2026-06-20 07:01:03 on dist-test-slave-mz20"
server_key: "a11ddec0b958d9496f0def15e4ee2eba"
server_key_iv: "a29a2a17afa6badd8c7b8a8a874fbbb4"
server_key_version: "encryptionkey@0"
I20260620 07:01:49.308707 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:49.327226 17184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:49.331364 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:49.331626 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:49.332698 17184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:49.334131 17184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:49.334229 17184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.334334 17184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:49.334410 17184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.364426 17184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38337
I20260620 07:01:49.364493 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38337 every 8 connection(s)
I20260620 07:01:49.365868 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:01:49.370371 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.368620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:45389)
0620 07:01:49.369331 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.369517 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
W20260620 07:01:49.371006 17313 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34755: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
I20260620 07:01:49.371802 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.371487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:54949)
0620 07:01:49.371630 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.371696 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 07:01:49.372563 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.372273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:53181)
0620 07:01:49.372378 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.372446 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 07:01:49.372772 17313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:01:49.375996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17184
I20260620 07:01:49.376569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43617
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34755
--webserver_port=41035
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:01:49.699409 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:01:49.699678 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:01:49.699770 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:01:49.707056 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:01:49.707171 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:01:49.707284 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:01:49.717183 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41035
--tserver_master_addrs=127.0.97.62:34755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:01:49.718437 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:01:49.719801 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:01:49.732575 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:49.732613 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:01:49.732645 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:01:49.733418 17316 server_base.cc:1034] running on GCE node
I20260620 07:01:49.733989 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:01:49.734921 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:01:49.736128 17316 hybrid_clock.cc:648] HybridClock initialized: now 1781938909736077 us; error 40 us; skew 500 ppm
I20260620 07:01:49.736413 17316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:01:49 dist-test-slave-mz20 krb5kdc[15861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:01:49.739591 17316 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:01:49.741019 17316 webserver.cc:460] Webserver started at http://127.0.97.3:41035/ using document root <none> and password file <none>
I20260620 07:01:49.741653 17316 fs_manager.cc:359] Metadata directory not provided
I20260620 07:01:49.741767 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:01:49.746881 17316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:49.749773 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.751191 17316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:01:49.751338 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "d3056e2070f445ad82bcc2e8311a24f5"
format_stamp: "Formatted at 2026-06-20 07:01:04 on dist-test-slave-mz20"
server_key: "05b28480c14dcc3a4e438abd0f90dc17"
server_key_iv: "445bf8e478a6ab3c28c0e683059ebf18"
server_key_version: "encryptionkey@0"
I20260620 07:01:49.752314 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:01:49.777858 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:01:49.781838 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:01:49.782089 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:01:49.783097 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:01:49.784458 17316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:01:49.784559 17316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.784663 17316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:01:49.784731 17316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:01:49.815800 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43617
I20260620 07:01:49.816009 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43617 every 8 connection(s)
I20260620 07:01:49.817239 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:01:49.818806 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17316
I20260620 07:01:49.822409 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.820671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:43485)
0620 07:01:49.821345 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.821563 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":255,"threads_started":1}
W20260620 07:01:49.823009 17444 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:34755: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
I20260620 07:01:49.823770 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.823478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:56099)
0620 07:01:49.823605 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.823669 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 07:01:49.824538 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.824250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:56561)
0620 07:01:49.824363 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.824438 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 07:01:49.824743 17444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 07:01:49.948027 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:49.947232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:33043)
0620 07:01:49.947715 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:49.947827 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":235,"threads_started":1}
I20260620 07:01:50.374321 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.373413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:38223)
0620 07:01:50.373981 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:50.374088 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":277,"threads_started":1}
I20260620 07:01:50.478193 17450 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:50.478590 17450 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:50.480480 17450 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
Jun 20 07:01:50 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Jun 20 07:01:50 dist-test-slave-mz20 krb5kdc[15861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:01:50.504774 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.481074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37059 (local address 127.0.97.3:43617)
0620 07:01:50.481855 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:50.481874 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:50.482183 (+ 309us) server_negotiation.cc:407] Connection header received
0620 07:01:50.484797 (+ 2614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:50.484812 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:50.485397 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:50.485801 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:50.488236 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.489528 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.491609 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.492364 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.492567 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:50.497352 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:50.497364 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:50.497392 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:50.497442 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:50.500687 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:50.501581 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:50.501595 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:50.501600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:50.501777 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:50.502268 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:50.502274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:50.502277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:50.502893 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:50.503060 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:50.503946 (+ 886us) server_negotiation.cc:299] Negotiation successful
0620 07:01:50.504359 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":325,"threads_started":1}
I20260620 07:01:50.504825 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.482692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38067 (local address 127.0.97.2:38337)
0620 07:01:50.483314 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 07:01:50.483326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:01:50.483366 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:01:50.485313 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:01:50.485327 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:01:50.486030 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:01:50.486476 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:01:50.488225 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.489484 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.491600 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.492311 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.492465 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:50.497280 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:01:50.497289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:01:50.497308 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:01:50.497369 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:01:50.500587 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:50.501583 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:50.501592 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:50.501599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:50.501775 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:01:50.502290 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:01:50.502293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:01:50.502296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:01:50.502940 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:01:50.503138 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 07:01:50.503947 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 07:01:50.504384 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":308,"threads_started":1}
I20260620 07:01:50.504861 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.480835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43617 (local address 127.0.97.1:37059)
0620 07:01:50.481955 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:50.481992 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:50.482799 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:50.486072 (+ 3273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:50.486087 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:50.487186 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:50.488017 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:50.488041 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.489719 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.489723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:50.491452 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:50.491468 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.492593 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.492604 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:50.492829 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:50.493634 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:50.493668 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:50.496916 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:50.500832 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:50.500836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:50.500839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:50.501424 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:50.501895 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:50.501899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:50.501902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:50.502146 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:50.503034 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:50.503043 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:50.503385 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 07:01:50.503653 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 07:01:50.504278 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":154,"spinlock_wait_cycles":29824,"thread_start_us":844,"threads_started":1}
I20260620 07:01:50.505224 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.482174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38337 (local address 127.0.97.1:38067)
0620 07:01:50.482828 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:50.482846 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:01:50.482960 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:01:50.486415 (+ 3455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:01:50.486418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:01:50.487163 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:01:50.488013 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:50.488036 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.489696 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.489701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:50.491450 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:01:50.491464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:01:50.492515 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:01:50.492522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:01:50.492834 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:01:50.493764 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:01:50.493782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:01:50.496916 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:01:50.500768 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:50.500778 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:50.500793 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:50.501420 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:50.501904 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:01:50.501908 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:01:50.501911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:01:50.502159 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:01:50.503081 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:01:50.503095 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:01:50.503367 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 07:01:50.503654 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 07:01:50.504497 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":395,"spinlock_wait_cycles":39168,"thread_start_us":254,"threads_started":1}
W20260620 07:01:50.508919 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
W20260620 07:01:50.509245 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:50.509357 17070 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbc9bdcc26924f98927cbaf74cad293f; no voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5
I20260620 07:01:50.509769 17450 raft_consensus.cc:2726] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:01:50.826249 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.825286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:54111)
0620 07:01:50.825810 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:50.825978 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":245,"threads_started":1}
I20260620 07:01:50.949497 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:50.949015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:54369)
0620 07:01:50.949205 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:50.949319 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 07:01:51.376366 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:51.375503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:50867)
0620 07:01:51.376061 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:51.376165 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
I20260620 07:01:51.828171 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:51.827279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:58861)
0620 07:01:51.827867 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:51.827967 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":349,"threads_started":1}
I20260620 07:01:51.951694 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:51.950660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:48889)
0620 07:01:51.951336 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:51.951463 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":322,"threads_started":1}
I20260620 07:01:52.378407 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:52.377457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:58089)
0620 07:01:52.378081 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:52.378190 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
I20260620 07:01:52.710265 17460 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:52.710503 17460 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:52.711285 17460 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
W20260620 07:01:52.712459 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
W20260620 07:01:52.712661 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:52.712826 17070 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbc9bdcc26924f98927cbaf74cad293f; no voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5
I20260620 07:01:52.713109 17460 raft_consensus.cc:2726] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:01:52.830154 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:52.829271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:50675)
0620 07:01:52.829815 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:52.829937 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":303,"threads_started":1}
I20260620 07:01:52.953550 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:52.952667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:44695)
0620 07:01:52.953199 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:52.953293 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
I20260620 07:01:53.380219 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:53.379351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:36987)
0620 07:01:53.379874 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:53.379982 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":267,"threads_started":1}
I20260620 07:01:53.831961 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:53.831152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:46067)
0620 07:01:53.831655 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:53.831743 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":241,"threads_started":1}
I20260620 07:01:53.955623 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:53.954754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:45589)
0620 07:01:53.955283 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:53.955373 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":293,"threads_started":1}
I20260620 07:01:54.382171 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:54.381374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:41915)
0620 07:01:54.381874 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:54.381969 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260620 07:01:54.695225 17467 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:54.695487 17467 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:54.696314 17467 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
W20260620 07:01:54.697463 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
W20260620 07:01:54.697695 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:54.697809 17070 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbc9bdcc26924f98927cbaf74cad293f; no voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5
I20260620 07:01:54.698081 17467 raft_consensus.cc:2726] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:01:54.834059 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:54.833118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:49599)
0620 07:01:54.833762 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:54.833864 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":354,"threads_started":1}
I20260620 07:01:54.957418 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:54.956564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:46245)
0620 07:01:54.957087 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:54.957183 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":288,"threads_started":1}
I20260620 07:01:55.384097 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:55.383209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:34415)
0620 07:01:55.383779 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:55.383867 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":332,"threads_started":1}
I20260620 07:01:55.835925 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:55.835095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:54119)
0620 07:01:55.835592 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:55.835696 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":271,"threads_started":1}
I20260620 07:01:55.959409 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:55.958510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:56481)
0620 07:01:55.959063 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:55.959160 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":274,"threads_started":1}
I20260620 07:01:56.386260 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:56.385291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:38617)
0620 07:01:56.385925 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:56.386026 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":271,"threads_started":1}
I20260620 07:01:56.838037 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:56.837028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:35989)
0620 07:01:56.837683 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:56.837805 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":283,"threads_started":1}
I20260620 07:01:56.961203 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:56.960381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:45871)
0620 07:01:56.960895 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:56.960997 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":283,"threads_started":1}
I20260620 07:01:57.388249 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:57.387407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:55379)
0620 07:01:57.387916 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:57.388030 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260620 07:01:57.504506 17477 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:01:57.504730 17477 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:01:57.505450 17477 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
W20260620 07:01:57.506657 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
W20260620 07:01:57.506892 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:01:57.506984 17070 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbc9bdcc26924f98927cbaf74cad293f; no voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5
I20260620 07:01:57.507277 17477 raft_consensus.cc:2726] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:01:57.839948 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:57.839154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:41273)
0620 07:01:57.839637 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:57.839729 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260620 07:01:57.963152 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:57.962209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:48819)
0620 07:01:57.962789 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:57.962883 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":312,"threads_started":1}
I20260620 07:01:58.390372 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:58.389423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:35367)
0620 07:01:58.390063 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:58.390167 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":363,"threads_started":1}
I20260620 07:01:58.841950 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:58.841095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:48659)
0620 07:01:58.841635 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:58.841749 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
I20260620 07:01:58.965157 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:58.964327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:46809)
0620 07:01:58.964850 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:58.964937 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":303,"threads_started":1}
I20260620 07:01:59.392343 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:59.391501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:47209)
0620 07:01:59.392042 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:59.392131 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":262,"threads_started":1}
I20260620 07:01:59.844297 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:59.843148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:59893)
0620 07:01:59.843878 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:59.843996 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":379,"threads_started":1}
I20260620 07:01:59.966951 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:01:59.966131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:35543)
0620 07:01:59.966641 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 07:01:59.966730 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":255,"threads_started":1}
I20260620 07:02:00.394280 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:00.393352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:46525)
0620 07:02:00.393965 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:00.394083 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
I20260620 07:02:00.846149 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:00.845347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:52093)
0620 07:02:00.845891 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:00.845981 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
I20260620 07:02:00.968714 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:00.967909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:40365)
0620 07:02:00.968440 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:00.968539 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":274,"threads_started":1}
I20260620 07:02:01.396150 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:01.395312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:44017)
0620 07:02:01.395836 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:01.395928 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260620 07:02:01.847863 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:01.847119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:49835)
0620 07:02:01.847556 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:01.847641 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260620 07:02:01.970453 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:01.969698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:33517)
0620 07:02:01.970190 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:01.970272 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":244,"threads_started":1}
I20260620 07:02:02.397898 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:02.397163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:60365)
0620 07:02:02.397657 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:02.397745 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260620 07:02:02.849455 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:02.848753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:50821)
0620 07:02:02.849195 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:02.849284 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260620 07:02:02.972114 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:02.971375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:34407)
0620 07:02:02.971841 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:02.971920 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260620 07:02:03.399523 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:03.398787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:49101)
0620 07:02:03.399238 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:03.399319 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260620 07:02:03.851176 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:03.850442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:45995)
0620 07:02:03.850910 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:03.850989 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":238,"threads_started":1}
I20260620 07:02:03.973977 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:03.973255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.1:45033)
0620 07:02:03.973663 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:03.973738 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":209,"threads_started":1}
I20260620 07:02:04.401348 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:04.400601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.2:58799)
0620 07:02:04.401093 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:04.401189 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260620 07:02:04.529356 17499 raft_consensus.cc:482] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:04.529578 17499 raft_consensus.cc:504] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3056e2070f445ad82bcc2e8311a24f5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43617 } } peers { permanent_uuid: "dbc9bdcc26924f98927cbaf74cad293f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33015 } } peers { permanent_uuid: "d1f59d78a6154bccabf708bb48244910" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38337 } }
I20260620 07:02:04.530292 17499 leader_election.cc:288] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617), d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337)
W20260620 07:02:04.531332 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d3056e2070f445ad82bcc2e8311a24f5 (127.0.97.3:43617): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
W20260620 07:02:04.531522 17070 leader_election.cc:341] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d1f59d78a6154bccabf708bb48244910 (127.0.97.2:38337): Not found: Tablet not found: 49535b19561d49a49a1ef0e9f2a88f3c
I20260620 07:02:04.531626 17070 leader_election.cc:302] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbc9bdcc26924f98927cbaf74cad293f; no voters: d1f59d78a6154bccabf708bb48244910, d3056e2070f445ad82bcc2e8311a24f5
I20260620 07:02:04.531867 17499 raft_consensus.cc:2726] T 49535b19561d49a49a1ef0e9f2a88f3c P dbc9bdcc26924f98927cbaf74cad293f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 07:02:04.829479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17011
I20260620 07:02:04.844667 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:04.830569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.0.97.1:33015)
0620 07:02:04.831199 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:04.831208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:04.831244 (+ 36us) server_negotiation.cc:407] Connection header received
0620 07:02:04.831331 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:04.831338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:04.831632 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:04.831794 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:04.833447 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:04.834455 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:04.835726 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:04.836296 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:04.836392 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:04.839045 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:04.839059 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:04.839075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:04.839110 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:04.841913 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:04.842599 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:04.842606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:04.842608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:04.842734 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:04.843037 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:04.843040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:04.843042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:04.843387 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:04.843690 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:04.844030 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 07:02:04.844292 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
I20260620 07:02:04.846050 17176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:04.852905 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:04.852177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34755 (local address 127.0.97.3:57941)
0620 07:02:04.852639 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:04.852725 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:34755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":245,"threads_started":1}
I20260620 07:02:04.935657 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17011
I20260620 07:02:04.950531 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17184
I20260620 07:02:04.963724 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:04.951300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.0.97.2:38337)
0620 07:02:04.951773 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:04.951778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:04.951795 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:02:04.951854 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:04.951858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:04.951940 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:04.952033 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:04.953304 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:04.954149 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:04.955444 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:04.955903 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:04.955996 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:04.958497 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:04.958502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:04.958505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:04.958564 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:04.961351 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:04.962051 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:04.962054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:04.962056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:04.962180 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:04.962488 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:04.962492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:04.962494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:04.962818 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:04.963020 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:04.963241 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 07:02:04.963429 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":271,"threads_started":1}
I20260620 07:02:04.964383 17308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:05.047829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17184
I20260620 07:02:05.063462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17316
I20260620 07:02:05.077253 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:05.064259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.97.3:43617)
0620 07:02:05.064831 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:05.064836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:05.064852 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:02:05.064907 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:05.064910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:05.064978 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:05.065074 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:05.066420 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:05.067268 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:05.068586 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:05.069041 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:05.069161 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:05.071994 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:05.072000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:05.072003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:05.072043 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:05.074643 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:05.075343 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:05.075349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:05.075352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:05.075493 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:05.075845 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:05.075849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:05.075851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:05.076120 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:05.076381 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:05.076645 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 07:02:05.076875 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":295,"threads_started":1}
I20260620 07:02:05.078040 17439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:05.161286 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17316
2026-06-20T07:02:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64348 ms)
[----------] 30 tests from SecurityITest (290344 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:05 dist-test-slave-mz20 krb5kdc[17509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:05 dist-test-slave-mz20 krb5kdc[17509](info): set up 2 sockets
Jun 20 07:02:05 dist-test-slave-mz20 krb5kdc[17509](info): commencing operation
krb5kdc: starting...
W20260620 07:02:06.186381 388 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:06 dist-test-slave-mz20 krb5kdc[17509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:06.419589 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44503
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:06.743132 17522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:06.743430 17522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:06.743546 17522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:06.750834 17522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:06.750950 17522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:06.751021 17522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:06.751106 17522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:06.751160 17522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:06.761255 17522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:06.762607 17522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:06.763897 17522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:06.775946 17527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:06.775919 17530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:06.775980 17528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:06.776741 17522 server_base.cc:1034] running on GCE node
I20260620 07:02:06.777522 17522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:06.778973 17522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:06.780184 17522 hybrid_clock.cc:648] HybridClock initialized: now 1781938926780143 us; error 29 us; skew 500 ppm
I20260620 07:02:06.780526 17522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:06 dist-test-slave-mz20 krb5kdc[17509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:06.787858 17522 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:06.789505 17522 webserver.cc:460] Webserver started at http://127.0.97.62:42713/ using document root <none> and password file <none>
I20260620 07:02:06.790200 17522 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:06.790321 17522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:06.790686 17522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:06.793676 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "dffa8f5a51b9479c9b7fcb23b52ca054"
format_stamp: "Formatted at 2026-06-20 07:02:06 on dist-test-slave-mz20"
server_key: "22bbeb33d94e3aac52f73a86f374033f"
server_key_iv: "8cf9b9e22db580d0b66a18e3aaff222b"
server_key_version: "encryptionkey@0"
I20260620 07:02:06.794421 17522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "dffa8f5a51b9479c9b7fcb23b52ca054"
format_stamp: "Formatted at 2026-06-20 07:02:06 on dist-test-slave-mz20"
server_key: "22bbeb33d94e3aac52f73a86f374033f"
server_key_iv: "8cf9b9e22db580d0b66a18e3aaff222b"
server_key_version: "encryptionkey@0"
I20260620 07:02:06.799716 17522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:02:06.803296 17537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:06.804665 17522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:06.804838 17522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "dffa8f5a51b9479c9b7fcb23b52ca054"
format_stamp: "Formatted at 2026-06-20 07:02:06 on dist-test-slave-mz20"
server_key: "22bbeb33d94e3aac52f73a86f374033f"
server_key_iv: "8cf9b9e22db580d0b66a18e3aaff222b"
server_key_version: "encryptionkey@0"
I20260620 07:02:06.805030 17522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:06.818594 17522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:06.822494 17522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:06.822781 17522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:06.839956 17522 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44503
I20260620 07:02:06.839978 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44503 every 8 connection(s)
I20260620 07:02:06.841320 17522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:06.841733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17522
I20260620 07:02:06.841928 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:06.842701 388 external_mini_cluster.cc:1428] Setting key 0891c119f364108678dd10acd95e2915
I20260620 07:02:06.845796 17589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:02:06 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:06.856364 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: Bootstrap starting.
I20260620 07:02:06.859712 17589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:06.860690 17589 log.cc:824] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:06.863844 17589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: No bootstrap required, opened a new log
I20260620 07:02:06.864154 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:06.844744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.0.97.62:44503)
0620 07:02:06.845511 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:06.845523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:06.845562 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:02:06.847218 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:06.847240 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:06.847591 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:06.847942 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:06.849296 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:06.850707 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:06.852013 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:06.852657 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:06.852809 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:06.856856 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:06.856864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:06.856880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:06.856917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:06.859939 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:06.860600 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:06.860607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:06.860612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:06.860785 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:06.861190 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:06.861199 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:06.861203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:06.861768 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:06.862061 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:06.862416 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 07:02:06.862869 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":358,"threads_started":1}
I20260620 07:02:06.871687 17589 raft_consensus.cc:348] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } }
I20260620 07:02:06.872017 17589 raft_consensus.cc:374] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:06.872092 17589 raft_consensus.cc:729] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dffa8f5a51b9479c9b7fcb23b52ca054, State: Initialized, Role: FOLLOWER
I20260620 07:02:06.872604 17589 consensus_queue.cc:260] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } }
I20260620 07:02:06.872807 17589 raft_consensus.cc:388] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:06.872900 17589 raft_consensus.cc:482] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:06.873027 17589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:06.875280 17589 raft_consensus.cc:504] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } }
I20260620 07:02:06.875737 17589 leader_election.cc:302] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dffa8f5a51b9479c9b7fcb23b52ca054; no voters:
I20260620 07:02:06.876230 17589 leader_election.cc:288] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:06.876363 17594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:06.876986 17594 raft_consensus.cc:686] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [term 1 LEADER]: Becoming Leader. State: Replica: dffa8f5a51b9479c9b7fcb23b52ca054, State: Running, Role: LEADER
I20260620 07:02:06.877493 17594 consensus_queue.cc:237] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } }
I20260620 07:02:06.878172 17589 sys_catalog.cc:564] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:06.880214 17596 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dffa8f5a51b9479c9b7fcb23b52ca054. Latest consensus state: current_term: 1 leader_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } } }
I20260620 07:02:06.880710 17596 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:06.882326 17600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:06.880225 17595 sys_catalog.cc:455] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dffa8f5a51b9479c9b7fcb23b52ca054" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44503 } } }
I20260620 07:02:06.882865 17595 sys_catalog.cc:458] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:06.887715 17600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:06.896263 17600 catalog_manager.cc:1347] Generated new cluster ID: 93f9a4696efd4459b5b86ec36dd6d557
I20260620 07:02:06.896402 17600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:06.911228 17600 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:06.912580 17600 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:06.918654 17600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: Generated new TSK 0
I20260620 07:02:06.919425 17600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:02:06.983232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:07.306222 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:07.306562 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:07.306705 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:07.314361 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:07.314484 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:07.314671 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:07.324664 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:07.326035 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:07.327397 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:07.340066 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:07.340013 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:07.340111 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:07.340821 17615 server_base.cc:1034] running on GCE node
I20260620 07:02:07.341405 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:07.342343 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:07.343556 17615 hybrid_clock.cc:648] HybridClock initialized: now 1781938927343499 us; error 42 us; skew 500 ppm
I20260620 07:02:07.343883 17615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:07 dist-test-slave-mz20 krb5kdc[17509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:07.346992 17615 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:02:07.348434 17615 webserver.cc:460] Webserver started at http://127.0.97.1:40031/ using document root <none> and password file <none>
I20260620 07:02:07.349067 17615 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:07.349162 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:07.349431 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:07.352460 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "7cf1832c10c54070abcbf50840ad3326"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "aa3883d9b500ef7a645604966aa70993"
server_key_iv: "e4fde87472ff42cc05789cc87ffbdd84"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.353214 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7cf1832c10c54070abcbf50840ad3326"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "aa3883d9b500ef7a645604966aa70993"
server_key_iv: "e4fde87472ff42cc05789cc87ffbdd84"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.358352 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:02:07.361934 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.363549 17615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:07.363698 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "7cf1832c10c54070abcbf50840ad3326"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "aa3883d9b500ef7a645604966aa70993"
server_key_iv: "e4fde87472ff42cc05789cc87ffbdd84"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.363895 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:07.382347 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:07.386763 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:07.387025 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:07.387969 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:07.389433 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:07.389525 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.389653 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:07.389707 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.420415 17615 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45173
I20260620 07:02:07.420456 17742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45173 every 8 connection(s)
I20260620 07:02:07.421729 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:07.425778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17615
I20260620 07:02:07.425956 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:07.426589 388 external_mini_cluster.cc:1428] Setting key 8012a9f39f2ac5504e7c2ebc408d23b9
Jun 20 07:02:07 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:07.441656 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:07.424119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32909 (local address 127.0.97.62:44503)
0620 07:02:07.424545 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:07.424551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:07.425471 (+ 920us) server_negotiation.cc:407] Connection header received
0620 07:02:07.427444 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:07.427449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:07.427534 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:07.427626 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:07.429227 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.429936 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.431550 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.431958 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.432078 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:07.436131 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:07.436136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:07.436140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:07.436177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:07.438599 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:07.439389 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:07.439392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:07.439394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:07.439508 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:07.439942 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:07.439946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:07.439948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:07.440222 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:07.440331 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:07.441203 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 07:02:07.441391 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":205,"threads_started":1}
I20260620 07:02:07.442917 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:07.424645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44503 (local address 127.0.97.1:32909)
0620 07:02:07.425317 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:07.425360 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:07.427156 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:07.427880 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:07.427888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:07.428499 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:07.429061 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:07.429077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.430100 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.430104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:07.431406 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:07.431420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.432201 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.432208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:07.432798 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:07.433468 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:07.433496 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:07.435886 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:07.438725 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:07.438734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:07.438752 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:07.439287 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:07.439592 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:07.439596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:07.439599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:07.439823 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:07.440336 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:07.440344 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:07.440764 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 07:02:07.441069 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 07:02:07.441582 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
I20260620 07:02:07.444548 17743 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44503
I20260620 07:02:07.444849 17743 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:07.445549 17743 heartbeater.cc:511] Master 127.0.97.62:44503 requested a full tablet report, sending...
I20260620 07:02:07.447671 17554 ts_manager.cc:194] Registered new tserver with Master: 7cf1832c10c54070abcbf50840ad3326 (127.0.97.1:45173)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 07:02:07.449258 17554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:02:07.478921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:07.794706 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:07.795011 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:07.795125 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:07.802613 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:07.802734 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:07.802889 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:07.813484 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:07.814848 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:07.816126 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:07.828876 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:07.828959 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:07.829181 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:07.830091 17749 server_base.cc:1034] running on GCE node
I20260620 07:02:07.830690 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:07.831601 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:07.832795 17749 hybrid_clock.cc:648] HybridClock initialized: now 1781938927832738 us; error 53 us; skew 500 ppm
I20260620 07:02:07.833091 17749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:07 dist-test-slave-mz20 krb5kdc[17509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:07.836009 17749 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:02:07.837345 17749 webserver.cc:460] Webserver started at http://127.0.97.2:41077/ using document root <none> and password file <none>
I20260620 07:02:07.837962 17749 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:07.838063 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:07.838317 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:07.841308 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "3e84976426644912adf2d4e6fae3fdf0"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "84bb7cee5c5e5a006803288547b1ed04"
server_key_iv: "8537cd1c1ee93c0c3d4bb8d6d7c9757c"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.842104 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3e84976426644912adf2d4e6fae3fdf0"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "84bb7cee5c5e5a006803288547b1ed04"
server_key_iv: "8537cd1c1ee93c0c3d4bb8d6d7c9757c"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.847400 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:02:07.851011 17764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.852453 17749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:07.852617 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "3e84976426644912adf2d4e6fae3fdf0"
format_stamp: "Formatted at 2026-06-20 07:02:07 on dist-test-slave-mz20"
server_key: "84bb7cee5c5e5a006803288547b1ed04"
server_key_iv: "8537cd1c1ee93c0c3d4bb8d6d7c9757c"
server_key_version: "encryptionkey@0"
I20260620 07:02:07.852813 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:07.879776 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:07.884318 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:07.884568 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:07.885529 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:07.886924 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:07.887003 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.887089 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:07.887130 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:07.917503 17749 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46479
I20260620 07:02:07.917552 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46479 every 8 connection(s)
I20260620 07:02:07.918762 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:07.921061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17749
I20260620 07:02:07.921216 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:07.921727 388 external_mini_cluster.cc:1428] Setting key ae9156c47674702a422902af6d9bc72e
Jun 20 07:02:07 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:07.939165 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:07.921289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60803 (local address 127.0.97.62:44503)
0620 07:02:07.921517 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:07.921524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:07.922780 (+ 1256us) server_negotiation.cc:407] Connection header received
0620 07:02:07.924631 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:07.924637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:07.924719 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:07.924818 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:07.926296 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.927237 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.928808 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.929362 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.929486 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:07.933401 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:07.933406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:07.933410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:07.933445 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:07.935980 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:07.936850 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:07.936854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:07.936856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:07.936995 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:07.937459 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:07.937462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:07.937464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:07.937742 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:07.937852 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:07.938723 (+ 871us) server_negotiation.cc:299] Negotiation successful
0620 07:02:07.938916 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:02:07.940629 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:07.921799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44503 (local address 127.0.97.2:60803)
0620 07:02:07.922632 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:07.922676 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:07.924355 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:07.925016 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:07.925023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:07.925563 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:07.926126 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:07.926143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.927405 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.927410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:07.928669 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:07.928681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:07.929597 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:07.929603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:07.929827 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:07.930507 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:07.930530 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:07.933145 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:07.936130 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:07.936139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:07.936159 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:07.936734 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:07.937097 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:07.937101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:07.937104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:07.937357 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:07.937854 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:07.937862 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:07.938312 (+ 450us) client_negotiation.cc:769] Sending connection context
0620 07:02:07.938641 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 07:02:07.939200 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":321,"threads_started":1}
I20260620 07:02:07.942133 17877 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44503
I20260620 07:02:07.942446 17877 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:07.943224 17877 heartbeater.cc:511] Master 127.0.97.62:44503 requested a full tablet report, sending...
I20260620 07:02:07.944712 17554 ts_manager.cc:194] Registered new tserver with Master: 3e84976426644912adf2d4e6fae3fdf0 (127.0.97.2:46479)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 07:02:07.945533 17554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:02:07.973645 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:08.303758 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:08.304050 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:08.304147 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:08.311604 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:08.311733 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:08.311851 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:08.321645 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:08.323024 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:08.324302 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:08.337513 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:08.337538 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:08.337476 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:08.338296 17882 server_base.cc:1034] running on GCE node
I20260620 07:02:08.338958 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:08.339953 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:08.341171 17882 hybrid_clock.cc:648] HybridClock initialized: now 1781938928341108 us; error 52 us; skew 500 ppm
I20260620 07:02:08.341502 17882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:08 dist-test-slave-mz20 krb5kdc[17509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:08.345098 17882 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:02:08.346683 17882 webserver.cc:460] Webserver started at http://127.0.97.3:32845/ using document root <none> and password file <none>
I20260620 07:02:08.347375 17882 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:08.347474 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:08.347757 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:08.350838 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "643b1703da1b4ef1b8b4fce7c193522e"
format_stamp: "Formatted at 2026-06-20 07:02:08 on dist-test-slave-mz20"
server_key: "f4570a8a900682b8359c36710e3273c0"
server_key_iv: "74d9f272ad53f75612b0bd989e99fd32"
server_key_version: "encryptionkey@0"
I20260620 07:02:08.351646 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "643b1703da1b4ef1b8b4fce7c193522e"
format_stamp: "Formatted at 2026-06-20 07:02:08 on dist-test-slave-mz20"
server_key: "f4570a8a900682b8359c36710e3273c0"
server_key_iv: "74d9f272ad53f75612b0bd989e99fd32"
server_key_version: "encryptionkey@0"
I20260620 07:02:08.357151 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:08.361001 17897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:08.362412 17882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 07:02:08.362599 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "643b1703da1b4ef1b8b4fce7c193522e"
format_stamp: "Formatted at 2026-06-20 07:02:08 on dist-test-slave-mz20"
server_key: "f4570a8a900682b8359c36710e3273c0"
server_key_iv: "74d9f272ad53f75612b0bd989e99fd32"
server_key_version: "encryptionkey@0"
I20260620 07:02:08.362839 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:08.376847 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:08.380856 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:08.381135 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:08.382052 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:08.383443 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:08.383522 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:08.383638 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:08.383693 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:08.412951 17882 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36557
I20260620 07:02:08.412992 18009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36557 every 8 connection(s)
I20260620 07:02:08.414153 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:08.416848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 17882
I20260620 07:02:08.417026 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:08.417572 388 external_mini_cluster.cc:1428] Setting key de7d20a0ba2ca8921fb61c5b241859ea
Jun 20 07:02:08 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:08.434801 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.416571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47547 (local address 127.0.97.62:44503)
0620 07:02:08.416736 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.416741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.418121 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 07:02:08.420194 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.420200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.420295 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.420387 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:08.422395 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.423193 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.425412 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.425811 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.425903 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.429146 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:08.429151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:08.429154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:08.429186 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:08.431684 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.432558 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.432563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.432566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.432752 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.433204 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.433207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.433209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.433465 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:08.433570 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.434268 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.434445 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:02:08.435614 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.417178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44503 (local address 127.0.97.3:47547)
0620 07:02:08.417975 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.418012 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.419914 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.420728 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.420740 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.421485 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:08.422209 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.422234 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.423356 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.423362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.425254 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.425269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.426010 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.426015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.426203 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.426879 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:08.426902 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:08.428954 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:08.431831 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:08.431840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:08.431856 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:08.432435 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:08.432864 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:08.432869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:08.432873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:08.433113 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:08.433554 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:08.433560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:08.433903 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.434123 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.434494 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":306,"threads_started":1}
I20260620 07:02:08.436877 18010 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44503
I20260620 07:02:08.437161 18010 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:08.437865 18010 heartbeater.cc:511] Master 127.0.97.62:44503 requested a full tablet report, sending...
I20260620 07:02:08.439301 17554 ts_manager.cc:194] Registered new tserver with Master: 643b1703da1b4ef1b8b4fce7c193522e (127.0.97.3:36557)
I20260620 07:02:08.440061 17554 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47547
I20260620 07:02:08.445266 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:08.452333 17743 heartbeater.cc:503] Master 127.0.97.62:44503 was elected leader, sending a full tablet report...
I20260620 07:02:08.461954 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.449630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.0.97.62:44503)
0620 07:02:08.449780 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.449785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.450240 (+ 455us) server_negotiation.cc:407] Connection header received
0620 07:02:08.450386 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.450390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.450454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.450567 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:08.451850 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.452608 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.454122 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.454526 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.454664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.456838 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:08.456843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:08.456847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:08.456877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:08.459404 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.460043 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.460049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.460053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.460177 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.460514 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.460518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.460522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.460846 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:08.461073 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.461431 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.461647 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:02:08.465615 17554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:08.472416 17554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:08.494766 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.484649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.0.97.3:36557)
0620 07:02:08.486666 (+ 2017us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.486684 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.486717 (+ 33us) server_negotiation.cc:407] Connection header received
0620 07:02:08.486809 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.486816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.487075 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.487270 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:08.488035 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.489780 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.492379 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.493552 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.493680 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.493908 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.494233 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.494400 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1833,"thread_start_us":1953,"threads_started":1}
I20260620 07:02:08.494764 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.484407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36557 (local address 127.0.0.1:52632)
0620 07:02:08.485321 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.485355 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.485526 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.487320 (+ 1794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.487329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.487369 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:08.487803 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.487815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.489932 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.489935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.492182 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.492197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.493760 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.493766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.493973 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.494025 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.494172 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.494357 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":544,"threads_started":1}
I20260620 07:02:08.496443 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.485319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.0.97.1:45173)
0620 07:02:08.486106 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.486115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.488106 (+ 1991us) server_negotiation.cc:407] Connection header received
0620 07:02:08.488223 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.488230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.488442 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.488850 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:08.490021 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.491578 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.494014 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.495427 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.495552 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.495789 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.495968 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.496130 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":472,"threads_started":1}
I20260620 07:02:08.497323 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.486739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.0.97.2:46479)
0620 07:02:08.488138 (+ 1399us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.488145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.490038 (+ 1893us) server_negotiation.cc:407] Connection header received
0620 07:02:08.490278 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.490283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.490437 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.490619 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:08.491174 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.492429 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.494792 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.496062 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.496151 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.496336 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.496861 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.497023 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":1171,"threads_started":1}
I20260620 07:02:08.498893 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.484782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45173 (local address 127.0.0.1:45740)
0620 07:02:08.487549 (+ 2767us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.487569 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.487703 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.488635 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.488639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.488663 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:08.489836 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.489850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.491726 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.491731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.493875 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.493889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.495619 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.495625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.495791 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.495826 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.495919 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.496046 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"thread_start_us":1466,"threads_started":1}
I20260620 07:02:08.498937 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.486395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46479 (local address 127.0.0.1:33906)
0620 07:02:08.489941 (+ 3546us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.489958 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.490121 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.490604 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.490609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.490638 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:08.491022 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.491030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.492594 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.492599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.494608 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.494623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.496503 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.496509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.496670 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.496700 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.497562 (+ 862us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.497692 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3405,"mutex_wait_us":27,"thread_start_us":3188,"threads_started":1}
I20260620 07:02:08.499009 17945 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0b990d91347e8910adb33ac68d0bb (DEFAULT_TABLE table=test-table [id=6cbb32e42f6340bd89ac6985c6af1940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:08.500442 17945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0b990d91347e8910adb33ac68d0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:08.502381 17678 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0b990d91347e8910adb33ac68d0bb (DEFAULT_TABLE table=test-table [id=6cbb32e42f6340bd89ac6985c6af1940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:08.502381 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0b990d91347e8910adb33ac68d0bb (DEFAULT_TABLE table=test-table [id=6cbb32e42f6340bd89ac6985c6af1940]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:08.503777 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0b990d91347e8910adb33ac68d0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:08.503777 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0b990d91347e8910adb33ac68d0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:08.511513 18028 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Bootstrap starting.
I20260620 07:02:08.514995 18028 tablet_bootstrap.cc:654] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:08.515990 18028 log.cc:824] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:08.517982 18030 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Bootstrap starting.
I20260620 07:02:08.519225 18028 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: No bootstrap required, opened a new log
I20260620 07:02:08.519500 18028 ts_tablet_manager.cc:1397] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:02:08.521590 18030 tablet_bootstrap.cc:654] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:08.522805 18030 log.cc:824] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:08.523233 18031 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Bootstrap starting.
I20260620 07:02:08.526134 18031 tablet_bootstrap.cc:654] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:08.526499 18030 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: No bootstrap required, opened a new log
I20260620 07:02:08.526829 18030 ts_tablet_manager.cc:1397] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260620 07:02:08.527128 18031 log.cc:824] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:08.529073 18028 raft_consensus.cc:348] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.529662 18028 raft_consensus.cc:374] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:08.529707 18031 tablet_bootstrap.cc:492] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: No bootstrap required, opened a new log
I20260620 07:02:08.529814 18028 raft_consensus.cc:729] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643b1703da1b4ef1b8b4fce7c193522e, State: Initialized, Role: FOLLOWER
I20260620 07:02:08.529999 18031 ts_tablet_manager.cc:1397] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 07:02:08.530609 18028 consensus_queue.cc:260] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.532388 18028 ts_tablet_manager.cc:1428] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260620 07:02:08.532599 18010 heartbeater.cc:503] Master 127.0.97.62:44503 was elected leader, sending a full tablet report...
I20260620 07:02:08.536156 18030 raft_consensus.cc:348] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.536597 18030 raft_consensus.cc:374] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:08.536711 18030 raft_consensus.cc:729] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf1832c10c54070abcbf50840ad3326, State: Initialized, Role: FOLLOWER
I20260620 07:02:08.537389 18030 consensus_queue.cc:260] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.539201 18030 ts_tablet_manager.cc:1428] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260620 07:02:08.539270 18031 raft_consensus.cc:348] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.539669 18031 raft_consensus.cc:374] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:08.539767 18031 raft_consensus.cc:729] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e84976426644912adf2d4e6fae3fdf0, State: Initialized, Role: FOLLOWER
I20260620 07:02:08.540441 18031 consensus_queue.cc:260] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.541492 17877 heartbeater.cc:503] Master 127.0.97.62:44503 was elected leader, sending a full tablet report...
I20260620 07:02:08.542115 18031 ts_tablet_manager.cc:1428] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260620 07:02:08.616163 18035 raft_consensus.cc:482] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:08.616449 18035 raft_consensus.cc:504] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.618172 18035 leader_election.cc:288] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 643b1703da1b4ef1b8b4fce7c193522e (127.0.97.3:36557), 3e84976426644912adf2d4e6fae3fdf0 (127.0.97.2:46479)
I20260620 07:02:08.624997 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.618791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44065 (local address 127.0.97.2:46479)
0620 07:02:08.619006 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.619013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.619488 (+ 475us) server_negotiation.cc:407] Connection header received
0620 07:02:08.619670 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.619676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.619745 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.619809 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:08.620472 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.621188 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.623095 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.623920 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.624055 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.624150 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.624629 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.624752 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:02:08.624996 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.618580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36199 (local address 127.0.97.3:36557)
0620 07:02:08.618778 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.618784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.619585 (+ 801us) server_negotiation.cc:407] Connection header received
0620 07:02:08.619752 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.619756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.619825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.619916 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:08.620512 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.621227 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.623106 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.623894 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.624017 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.624133 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.624591 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.624733 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:02:08.625805 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.618562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36557 (local address 127.0.97.1:36199)
0620 07:02:08.619457 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.619475 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.619604 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.619945 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.619948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.619978 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:08.620339 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.620348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.621358 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.621362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.622953 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.622965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.624161 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.624170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.624400 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.624434 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.625188 (+ 754us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.625354 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":92,"thread_start_us":427,"threads_started":1}
I20260620 07:02:08.625825 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.618504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46479 (local address 127.0.97.1:44065)
0620 07:02:08.619377 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:08.619396 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:08.619526 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:08.619845 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:08.619850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:08.619872 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:08.620327 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.620338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.621314 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.621319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.622951 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:08.622964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.624287 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.624293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:08.624459 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.624484 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:02:08.625188 (+ 704us) client_negotiation.cc:240] Negotiation successful
0620 07:02:08.625363 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":29,"thread_start_us":636,"threads_started":1}
I20260620 07:02:08.627007 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0b990d91347e8910adb33ac68d0bb" candidate_uuid: "7cf1832c10c54070abcbf50840ad3326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e84976426644912adf2d4e6fae3fdf0" is_pre_election: true
I20260620 07:02:08.627012 17965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0b990d91347e8910adb33ac68d0bb" candidate_uuid: "7cf1832c10c54070abcbf50840ad3326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643b1703da1b4ef1b8b4fce7c193522e" is_pre_election: true
I20260620 07:02:08.627592 17965 raft_consensus.cc:2454] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cf1832c10c54070abcbf50840ad3326 in term 0.
I20260620 07:02:08.627592 17832 raft_consensus.cc:2454] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cf1832c10c54070abcbf50840ad3326 in term 0.
I20260620 07:02:08.628317 17633 leader_election.cc:302] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643b1703da1b4ef1b8b4fce7c193522e, 7cf1832c10c54070abcbf50840ad3326; no voters:
I20260620 07:02:08.628677 18035 raft_consensus.cc:2781] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:08.628832 18035 raft_consensus.cc:482] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:08.628931 18035 raft_consensus.cc:3037] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:08.631174 18035 raft_consensus.cc:504] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.631771 18035 leader_election.cc:288] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [CANDIDATE]: Term 1 election: Requested vote from peers 643b1703da1b4ef1b8b4fce7c193522e (127.0.97.3:36557), 3e84976426644912adf2d4e6fae3fdf0 (127.0.97.2:46479)
I20260620 07:02:08.632262 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0b990d91347e8910adb33ac68d0bb" candidate_uuid: "7cf1832c10c54070abcbf50840ad3326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e84976426644912adf2d4e6fae3fdf0"
I20260620 07:02:08.632296 17965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0b990d91347e8910adb33ac68d0bb" candidate_uuid: "7cf1832c10c54070abcbf50840ad3326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643b1703da1b4ef1b8b4fce7c193522e"
I20260620 07:02:08.632443 17832 raft_consensus.cc:3037] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:08.632488 17965 raft_consensus.cc:3037] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:08.634685 17965 raft_consensus.cc:2454] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cf1832c10c54070abcbf50840ad3326 in term 1.
I20260620 07:02:08.634689 17832 raft_consensus.cc:2454] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cf1832c10c54070abcbf50840ad3326 in term 1.
I20260620 07:02:08.635095 17633 leader_election.cc:302] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643b1703da1b4ef1b8b4fce7c193522e, 7cf1832c10c54070abcbf50840ad3326; no voters:
I20260620 07:02:08.635376 18035 raft_consensus.cc:2781] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:08.635782 18035 raft_consensus.cc:686] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 1 LEADER]: Becoming Leader. State: Replica: 7cf1832c10c54070abcbf50840ad3326, State: Running, Role: LEADER
I20260620 07:02:08.636310 18035 consensus_queue.cc:237] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } }
I20260620 07:02:08.640683 17554 catalog_manager.cc:5495] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cf1832c10c54070abcbf50840ad3326 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7cf1832c10c54070abcbf50840ad3326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cf1832c10c54070abcbf50840ad3326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 } health_report { overall_health: UNKNOWN } } }
W20260620 07:02:08.665755 18011 tablet.cc:2369] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:02:08.670787 17878 tablet.cc:2369] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:02:08.673971 17744 tablet.cc:2369] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:08.692281 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.687183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.0.97.1:45173)
0620 07:02:08.687398 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.687402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.687417 (+ 15us) server_negotiation.cc:407] Connection header received
0620 07:02:08.687504 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.687507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.687623 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.687755 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:08.688129 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.688846 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.690200 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.690626 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.690737 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.691208 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:08.691718 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:08.691848 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.691948 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.692030 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:02:08.700840 17965 raft_consensus.cc:1261] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 1 FOLLOWER]: Refusing update from remote peer 7cf1832c10c54070abcbf50840ad3326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:08.700842 17832 raft_consensus.cc:1261] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 1 FOLLOWER]: Refusing update from remote peer 7cf1832c10c54070abcbf50840ad3326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:08.701788 18039 consensus_queue.cc:1035] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643b1703da1b4ef1b8b4fce7c193522e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:08.702114 18035 consensus_queue.cc:1035] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e84976426644912adf2d4e6fae3fdf0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:08.713183 18042 mvcc.cc:205] Tried to move back new op lower bound from 7298821851949047808 to 7298821851698446336. Current Snapshot: MvccSnapshot[applied={T|T < 7298821851949047808}]
I20260620 07:02:08.726262 17554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:08.726617 17554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:08.730401 17554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dffa8f5a51b9479c9b7fcb23b52ca054: Sending DeleteTablet for 3 replicas of tablet 39c0b990d91347e8910adb33ac68d0bb
I20260620 07:02:08.731644 17945 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0b990d91347e8910adb33ac68d0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52632
I20260620 07:02:08.731657 17678 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0b990d91347e8910adb33ac68d0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45740
I20260620 07:02:08.731972 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0b990d91347e8910adb33ac68d0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33906
I20260620 07:02:08.733489 18053 tablet_replica.cc:330] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: stopping tablet replica
I20260620 07:02:08.733554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17615
I20260620 07:02:08.733970 18053 raft_consensus.cc:2229] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:08.734464 18053 raft_consensus.cc:2258] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:08.736212 18052 tablet_replica.cc:330] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: stopping tablet replica
I20260620 07:02:08.736490 18054 tablet_replica.cc:330] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: stopping tablet replica
I20260620 07:02:08.736914 18052 raft_consensus.cc:2229] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:08.736965 18053 ts_tablet_manager.cc:1905] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:08.737375 18054 raft_consensus.cc:2229] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:08.738067 18052 raft_consensus.cc:2258] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:08.738605 18054 raft_consensus.cc:2258] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:08.740990 18054 ts_tablet_manager.cc:1905] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:08.741010 18052 ts_tablet_manager.cc:1905] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:08.743783 18053 ts_tablet_manager.cc:1918] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:08.743970 18053 log.cc:1196] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/39c0b990d91347e8910adb33ac68d0bb
I20260620 07:02:08.744546 18053 ts_tablet_manager.cc:1939] T 39c0b990d91347e8910adb33ac68d0bb P 643b1703da1b4ef1b8b4fce7c193522e: Deleting consensus metadata
I20260620 07:02:08.746726 17540 catalog_manager.cc:4841] TS 643b1703da1b4ef1b8b4fce7c193522e (127.0.97.3:36557): tablet 39c0b990d91347e8910adb33ac68d0bb (table test-table [id=6cbb32e42f6340bd89ac6985c6af1940]) successfully deleted
I20260620 07:02:08.747913 18054 ts_tablet_manager.cc:1918] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:08.747918 18052 ts_tablet_manager.cc:1918] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:08.748065 18052 log.cc:1196] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/39c0b990d91347e8910adb33ac68d0bb
I20260620 07:02:08.748183 18054 log.cc:1196] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/39c0b990d91347e8910adb33ac68d0bb
I20260620 07:02:08.748685 18052 ts_tablet_manager.cc:1939] T 39c0b990d91347e8910adb33ac68d0bb P 7cf1832c10c54070abcbf50840ad3326: Deleting consensus metadata
I20260620 07:02:08.748772 18054 ts_tablet_manager.cc:1939] T 39c0b990d91347e8910adb33ac68d0bb P 3e84976426644912adf2d4e6fae3fdf0: Deleting consensus metadata
Jun 20 07:02:08 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:02:08.750610 17539 catalog_manager.cc:4841] TS 3e84976426644912adf2d4e6fae3fdf0 (127.0.97.2:46479): tablet 39c0b990d91347e8910adb33ac68d0bb (table test-table [id=6cbb32e42f6340bd89ac6985c6af1940]) successfully deleted
I20260620 07:02:08.750636 17541 catalog_manager.cc:4841] TS 7cf1832c10c54070abcbf50840ad3326 (127.0.97.1:45173): tablet 39c0b990d91347e8910adb33ac68d0bb (table test-table [id=6cbb32e42f6340bd89ac6985c6af1940]) successfully deleted
I20260620 07:02:08.757602 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.735263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.97.1:45173)
0620 07:02:08.735697 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.735703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.739611 (+ 3908us) server_negotiation.cc:407] Connection header received
0620 07:02:08.739889 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.739894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.739980 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.740057 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:08.741766 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.742909 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.744700 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.745355 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.745450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.751806 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:08.751818 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:08.751835 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:08.751881 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:08.754821 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.755534 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.755540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.755543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.755679 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.756041 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.756045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.756047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.756429 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:08.756732 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.756995 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.757234 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317}
I20260620 07:02:08.758302 17738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:08.855772 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17615
I20260620 07:02:08.870399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17749
Jun 20 07:02:08 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:02:08.884752 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.871124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.0.97.2:46479)
0620 07:02:08.871362 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.871368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.871391 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:02:08.871586 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.871590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.871654 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.871742 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:08.872950 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.873656 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.875111 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.875499 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.875607 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.879615 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:08.879628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:08.879643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:08.879683 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:08.882145 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.882894 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.882902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.882904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.883041 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.883364 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.883367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.883369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.883748 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:08.884025 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:08.884243 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:02:08.884414 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:02:08.885353 17872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:08.972671 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17749
I20260620 07:02:08.986382 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17882
Jun 20 07:02:08 dist-test-slave-mz20 krb5kdc[17509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:02:09.000833 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:08.987189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.0.97.3:36557)
0620 07:02:08.987378 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:08.987384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:08.987404 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:02:08.987532 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:08.987538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:08.987610 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:08.987703 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:08.988882 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.989624 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.990849 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:08.991244 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:08.991351 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:08.995463 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:08.995480 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:08.995497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:08.995532 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:08.998114 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.998925 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.998932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.998934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.999069 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:08.999461 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:08.999464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:08.999467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:08.999863 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:09.000103 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:09.000389 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 07:02:09.000575 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:02:09.001485 18005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:09.089920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17882
I20260620 07:02:09.103559 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 17522
I20260620 07:02:09.104667 17584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:09.213603 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 17522
2026-06-20T07:02:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4053 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T07:02:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:09Z Disabled control of system clock
I20260620 07:02:09.282629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:09.608063 18063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:09.608408 18063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:09.608541 18063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:09.615943 18063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:09.616071 18063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:09.616138 18063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:09.616225 18063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:09.616286 18063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:09.625515 18063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:09.626842 18063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:09.628574 18063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:09.639809 18068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:09.639783 18069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:09.639750 18071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:09.640425 18063 server_base.cc:1034] running on GCE node
I20260620 07:02:09.641136 18063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:09.642477 18063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:09.643698 18063 hybrid_clock.cc:648] HybridClock initialized: now 1781938929643656 us; error 34 us; skew 500 ppm
I20260620 07:02:09.644037 18063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:09.646306 18063 webserver.cc:460] Webserver started at http://127.0.97.62:35853/ using document root <none> and password file <none>
I20260620 07:02:09.647001 18063 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:09.647094 18063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:09.647403 18063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:09.650312 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "2096f78513054c12a9acc85c127cf257"
format_stamp: "Formatted at 2026-06-20 07:02:09 on dist-test-slave-mz20"
server_key: "bf3af652516abefc5f6c3faa6d13c7e9"
server_key_iv: "08437b01424b1803e07aabfbb6a9463c"
server_key_version: "encryptionkey@0"
I20260620 07:02:09.651178 18063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "2096f78513054c12a9acc85c127cf257"
format_stamp: "Formatted at 2026-06-20 07:02:09 on dist-test-slave-mz20"
server_key: "bf3af652516abefc5f6c3faa6d13c7e9"
server_key_iv: "08437b01424b1803e07aabfbb6a9463c"
server_key_version: "encryptionkey@0"
I20260620 07:02:09.656370 18063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:09.659891 18077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:09.661301 18063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:02:09.661474 18063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "2096f78513054c12a9acc85c127cf257"
format_stamp: "Formatted at 2026-06-20 07:02:09 on dist-test-slave-mz20"
server_key: "bf3af652516abefc5f6c3faa6d13c7e9"
server_key_iv: "08437b01424b1803e07aabfbb6a9463c"
server_key_version: "encryptionkey@0"
I20260620 07:02:09.661679 18063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:09.685154 18063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:09.686017 18063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:09.686292 18063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:09.705466 18063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39897
I20260620 07:02:09.705473 18128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39897 every 8 connection(s)
I20260620 07:02:09.706835 18063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:09.710575 18129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:09.714417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18063
I20260620 07:02:09.714602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:09.715070 388 external_mini_cluster.cc:1428] Setting key 9510dc787b4094d6754615804739edc3
I20260620 07:02:09.721925 18129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: Bootstrap starting.
I20260620 07:02:09.725648 18129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:09.726899 18129 log.cc:824] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:09.729743 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:09.717078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.0.97.62:39897)
0620 07:02:09.718058 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:09.718070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:09.718126 (+ 56us) server_negotiation.cc:407] Connection header received
0620 07:02:09.720303 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:09.720331 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:09.720848 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:09.721221 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:09.721958 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:09.723372 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:09.725345 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:09.726079 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:09.726244 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:09.727492 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:09.727500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:09.727514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:09.728082 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:09.728167 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:09.728189 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:09.728291 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:09.728515 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 07:02:09.728815 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":337,"threads_started":1}
I20260620 07:02:09.729827 18129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: No bootstrap required, opened a new log
I20260620 07:02:09.737727 18129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260620 07:02:09.738154 18129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:09.738253 18129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2096f78513054c12a9acc85c127cf257, State: Initialized, Role: FOLLOWER
I20260620 07:02:09.738870 18129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260620 07:02:09.739105 18129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:09.739249 18129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:09.739432 18129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:09.741732 18129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260620 07:02:09.742239 18129 leader_election.cc:302] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2096f78513054c12a9acc85c127cf257; no voters:
I20260620 07:02:09.742833 18129 leader_election.cc:288] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:09.742978 18134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:09.743924 18134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [term 1 LEADER]: Becoming Leader. State: Replica: 2096f78513054c12a9acc85c127cf257, State: Running, Role: LEADER
I20260620 07:02:09.744463 18134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260620 07:02:09.744968 18129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:09.747336 18136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2096f78513054c12a9acc85c127cf257. Latest consensus state: current_term: 1 leader_uuid: "2096f78513054c12a9acc85c127cf257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } }
I20260620 07:02:09.747757 18136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:09.748299 18135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2096f78513054c12a9acc85c127cf257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2096f78513054c12a9acc85c127cf257" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } }
I20260620 07:02:09.750844 18135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:09.752168 18143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:09.757328 18143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:09.765784 18143 catalog_manager.cc:1347] Generated new cluster ID: 0922827fbbd44e6a9243dfab6fa56394
I20260620 07:02:09.765913 18143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:09.794320 18143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:09.795617 18143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:09.802424 18143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: Generated new TSK 0
I20260620 07:02:09.803484 18143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:09.815673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:10.146355 18153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:10.146782 18153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:10.146919 18153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:10.154515 18153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:10.154663 18153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:10.154886 18153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:10.164949 18153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:10.166235 18153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:10.168056 18153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:10.180876 18158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:10.180855 18161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:10.180820 18159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:10.181586 18153 server_base.cc:1034] running on GCE node
I20260620 07:02:10.182237 18153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:10.183292 18153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:10.184525 18153 hybrid_clock.cc:648] HybridClock initialized: now 1781938930184471 us; error 45 us; skew 500 ppm
I20260620 07:02:10.184850 18153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:10.187227 18153 webserver.cc:460] Webserver started at http://127.0.97.1:39581/ using document root <none> and password file <none>
I20260620 07:02:10.187956 18153 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:10.188063 18153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:10.188381 18153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:10.191495 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "5da9927b8153418a9d3ae821cc732460"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "3d8e0fc6de650ef57197ea2e549885c9"
server_key_iv: "13d3ef4890e0c100fba7f3f97b50923f"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.192313 18153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5da9927b8153418a9d3ae821cc732460"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "3d8e0fc6de650ef57197ea2e549885c9"
server_key_iv: "13d3ef4890e0c100fba7f3f97b50923f"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.197607 18153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:10.201308 18167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.202970 18153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:02:10.203135 18153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "5da9927b8153418a9d3ae821cc732460"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "3d8e0fc6de650ef57197ea2e549885c9"
server_key_iv: "13d3ef4890e0c100fba7f3f97b50923f"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.203337 18153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:10.245579 18153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:10.246469 18153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:10.246744 18153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:10.247699 18153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:10.249207 18153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:10.249296 18153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.249410 18153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:10.249485 18153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.279294 18153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33245
I20260620 07:02:10.279349 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33245 every 8 connection(s)
I20260620 07:02:10.280571 18153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:10.282044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18153
I20260620 07:02:10.286399 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:10.287031 388 external_mini_cluster.cc:1428] Setting key 17a425ecf44f24df5bbdc0047eb2afe3
I20260620 07:02:10.289878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:10.294478 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:10.283128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60811 (local address 127.0.97.62:39897)
0620 07:02:10.283596 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:10.283603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:10.284975 (+ 1372us) server_negotiation.cc:407] Connection header received
0620 07:02:10.287591 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:10.287598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:10.287798 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:10.287914 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:10.289230 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.289948 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.291605 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.292140 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.292261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:10.293560 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:10.293604 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:10.293608 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:10.293639 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:10.293686 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:10.293702 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:10.293786 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:10.294000 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 07:02:10.294099 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":214,"threads_started":1}
I20260620 07:02:10.295409 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:10.283685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.1:60811)
0620 07:02:10.284765 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:10.284818 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:10.287241 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:10.288195 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:10.288205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:10.288304 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:10.288987 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:10.289014 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.290153 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.290157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:10.291448 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:10.291462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.292356 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.292361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:10.292557 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:10.293226 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:10.293229 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:10.293269 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:10.293352 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:10.293806 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:10.293816 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:10.293823 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:02:10.293970 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:02:10.294281 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":278,"threads_started":1}
I20260620 07:02:10.296993 18280 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260620 07:02:10.297292 18280 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:10.298046 18280 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260620 07:02:10.300231 18094 ts_manager.cc:194] Registered new tserver with Master: 5da9927b8153418a9d3ae821cc732460 (127.0.97.1:33245)
I20260620 07:02:10.302255 18094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60811
W20260620 07:02:10.608937 18284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:10.609237 18284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:10.609324 18284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:10.616636 18284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:10.616739 18284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:10.616844 18284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:10.625829 18284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:10.627060 18284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:10.628715 18284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:10.641151 18289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:10.641131 18290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:10.641093 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:10.642016 18284 server_base.cc:1034] running on GCE node
I20260620 07:02:10.642688 18284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:10.643663 18284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:10.644856 18284 hybrid_clock.cc:648] HybridClock initialized: now 1781938930644806 us; error 40 us; skew 500 ppm
I20260620 07:02:10.645157 18284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:10.647383 18284 webserver.cc:460] Webserver started at http://127.0.97.2:34133/ using document root <none> and password file <none>
I20260620 07:02:10.648033 18284 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:10.648131 18284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:10.648440 18284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:10.651376 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "a7d8a300fa5b494d910fa6ad0df0a04e"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "c890e91ef077d177b3b4b34bf61e0e14"
server_key_iv: "fb3e22d4d3aead5ea99226af87cf728d"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.652186 18284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a7d8a300fa5b494d910fa6ad0df0a04e"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "c890e91ef077d177b3b4b34bf61e0e14"
server_key_iv: "fb3e22d4d3aead5ea99226af87cf728d"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.657356 18284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:10.660702 18298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.661927 18284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 07:02:10.662077 18284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "a7d8a300fa5b494d910fa6ad0df0a04e"
format_stamp: "Formatted at 2026-06-20 07:02:10 on dist-test-slave-mz20"
server_key: "c890e91ef077d177b3b4b34bf61e0e14"
server_key_iv: "fb3e22d4d3aead5ea99226af87cf728d"
server_key_version: "encryptionkey@0"
I20260620 07:02:10.662272 18284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:10.692677 18284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:10.693562 18284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:10.693845 18284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:10.694873 18284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:10.696326 18284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:10.696444 18284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.696566 18284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:10.696640 18284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:10.731604 18284 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34823
I20260620 07:02:10.731669 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34823 every 8 connection(s)
I20260620 07:02:10.732825 18284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:10.736598 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18284
I20260620 07:02:10.736768 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:10.737310 388 external_mini_cluster.cc:1428] Setting key e2bac334da5dfb5d999e9961dc34243e
I20260620 07:02:10.740077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:10.744921 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:10.735144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54887 (local address 127.0.97.62:39897)
0620 07:02:10.735369 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:10.735376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:10.736795 (+ 1419us) server_negotiation.cc:407] Connection header received
0620 07:02:10.738922 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:10.738927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:10.739004 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:10.739087 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:10.740118 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.740886 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.742351 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.742787 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.742898 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:10.744139 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:10.744144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:10.744147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:10.744165 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:10.744194 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:10.744206 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:10.744280 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:10.744476 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 07:02:10.744583 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:02:10.745888 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:10.735676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.2:54887)
0620 07:02:10.736624 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:10.736671 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:10.738672 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:10.739311 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:10.739319 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:10.739418 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:10.739960 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:10.739976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.741031 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.741035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:10.742218 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:10.742229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:10.742996 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:10.743003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:10.743207 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:10.743851 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:10.743853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:10.743895 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:10.743965 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:10.744288 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:10.744299 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:10.744306 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:02:10.744454 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:02:10.744841 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":256,"threads_started":1}
I20260620 07:02:10.747467 18411 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260620 07:02:10.747762 18411 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:10.748502 18411 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260620 07:02:10.749944 18094 ts_manager.cc:194] Registered new tserver with Master: a7d8a300fa5b494d910fa6ad0df0a04e (127.0.97.2:34823)
I20260620 07:02:10.750972 18094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:54887
W20260620 07:02:11.056244 18414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:11.056516 18414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:11.056597 18414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:11.063773 18414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:11.063876 18414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:11.063982 18414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:11.073072 18414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:11.074292 18414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:11.075971 18414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:11.088899 18420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:11.088935 18419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:11.088913 18422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:11.089634 18414 server_base.cc:1034] running on GCE node
I20260620 07:02:11.090317 18414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:11.091377 18414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:11.092609 18414 hybrid_clock.cc:648] HybridClock initialized: now 1781938931092563 us; error 43 us; skew 500 ppm
I20260620 07:02:11.092944 18414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:11.095213 18414 webserver.cc:460] Webserver started at http://127.0.97.3:38155/ using document root <none> and password file <none>
I20260620 07:02:11.095868 18414 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:11.095963 18414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:11.096261 18414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:11.099081 18414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "f33e406f666e4733a28c8300d2449a2c"
format_stamp: "Formatted at 2026-06-20 07:02:11 on dist-test-slave-mz20"
server_key: "81a8943a8e128241720c3b1ab6e2237d"
server_key_iv: "c532e30f0cc466c9535b31518eb1dba2"
server_key_version: "encryptionkey@0"
I20260620 07:02:11.099845 18414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f33e406f666e4733a28c8300d2449a2c"
format_stamp: "Formatted at 2026-06-20 07:02:11 on dist-test-slave-mz20"
server_key: "81a8943a8e128241720c3b1ab6e2237d"
server_key_iv: "c532e30f0cc466c9535b31518eb1dba2"
server_key_version: "encryptionkey@0"
I20260620 07:02:11.104802 18414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:11.108247 18428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:11.109664 18414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:02:11.109833 18414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "f33e406f666e4733a28c8300d2449a2c"
format_stamp: "Formatted at 2026-06-20 07:02:11 on dist-test-slave-mz20"
server_key: "81a8943a8e128241720c3b1ab6e2237d"
server_key_iv: "c532e30f0cc466c9535b31518eb1dba2"
server_key_version: "encryptionkey@0"
I20260620 07:02:11.110054 18414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:11.129061 18414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:11.129900 18414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:11.130174 18414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:11.131160 18414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:11.132741 18414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:11.132850 18414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:11.132956 18414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:11.133024 18414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:11.162629 18414 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35951
I20260620 07:02:11.162696 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35951 every 8 connection(s)
I20260620 07:02:11.163836 18414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:11.174151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18414
I20260620 07:02:11.174327 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:11.174880 388 external_mini_cluster.cc:1428] Setting key ab82be10a438a86b582611309cc80957
I20260620 07:02:11.175875 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.166426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53027 (local address 127.0.97.62:39897)
0620 07:02:11.166643 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.166649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.167930 (+ 1281us) server_negotiation.cc:407] Connection header received
0620 07:02:11.169612 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.169617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.169684 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.169768 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:11.170789 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.171593 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.173125 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.173542 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.173639 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.175120 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:11.175125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:11.175128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:11.175146 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:11.175178 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:11.175189 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:11.175258 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.175515 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.175602 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:02:11.177012 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.166965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.3:53027)
0620 07:02:11.167808 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.167841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.169392 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.169945 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.169952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.170032 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:11.170638 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.170659 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.171772 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.171776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.172979 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.172991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.173729 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.173735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.174006 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.174759 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:11.174763 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:11.174812 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:11.174915 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:11.175282 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:11.175293 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:11.175301 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.175449 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.175803 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260620 07:02:11.178301 18541 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260620 07:02:11.178611 18541 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:11.179314 18541 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260620 07:02:11.180591 18094 ts_manager.cc:194] Registered new tserver with Master: f33e406f666e4733a28c8300d2449a2c (127.0.97.3:35951)
I20260620 07:02:11.181375 18094 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53027
I20260620 07:02:11.189944 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:11.199242 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.193736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.0.97.62:39897)
0620 07:02:11.193983 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.193990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.194260 (+ 270us) server_negotiation.cc:407] Connection header received
0620 07:02:11.194390 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.194394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.194480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.194589 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:11.195027 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.195720 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.197053 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.197462 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.197558 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.198505 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:11.198509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:11.198513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:11.198529 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:11.198595 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:11.198608 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:11.198674 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.198865 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.198944 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:02:11.202898 18094 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:11.209784 18094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:11.246874 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.222579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.97.1:33245)
0620 07:02:11.239409 (+ 16830us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.239419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.239449 (+ 30us) server_negotiation.cc:407] Connection header received
0620 07:02:11.239549 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.239557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.239800 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.240207 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:11.240752 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.242308 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.244564 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.245647 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.245812 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.246075 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.246343 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.246519 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16649,"thread_start_us":16539,"threads_started":1}
I20260620 07:02:11.246886 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.222057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47422)
0620 07:02:11.238461 (+ 16404us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.238482 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.238845 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.240051 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.240057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.240114 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:11.240536 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.240551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.242465 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.242471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.244377 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.244390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.245881 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.245888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.246096 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.246151 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.246273 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.246476 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16273,"mutex_wait_us":21,"spinlock_wait_cycles":4352,"thread_start_us":15035,"threads_started":1}
I20260620 07:02:11.248488 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.221538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34823 (local address 127.0.0.1:49366)
0620 07:02:11.237413 (+ 15875us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.237436 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.237571 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.240047 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.240056 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.240096 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:11.241129 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.241141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.242905 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.242910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.245259 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.245278 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.247673 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.247679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.247839 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.247870 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.247962 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.248142 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15694,"mutex_wait_us":47,"spinlock_wait_cycles":3840,"thread_start_us":10169,"threads_started":1}
I20260620 07:02:11.249059 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.222108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.97.2:34823)
0620 07:02:11.239008 (+ 16900us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.239018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.239057 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:02:11.239168 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.239175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.239493 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.239708 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:11.241319 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.242767 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.246004 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.247485 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.247613 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.247825 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.248021 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.248204 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16687,"thread_start_us":18479,"threads_started":1}
I20260620 07:02:11.249243 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.221467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35951 (local address 127.0.0.1:60536)
0620 07:02:11.232224 (+ 10757us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.232280 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.232494 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.241275 (+ 8781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.241279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.241304 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:11.241603 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.241613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.243134 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.243139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.245296 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.245307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.248562 (+ 3255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.248569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.248731 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.248762 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.248859 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.248986 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10417,"mutex_wait_us":1,"thread_start_us":17089,"threads_started":1}
I20260620 07:02:11.249927 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.221592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.97.3:35951)
0620 07:02:11.240843 (+ 19251us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.240849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.240874 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:02:11.240944 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.240949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.241098 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.241250 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:11.241787 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.242998 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.245494 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.248359 (+ 2865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.248497 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.248747 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.249518 (+ 771us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.249658 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19109,"thread_start_us":18953,"threads_started":1}
I20260620 07:02:11.251484 18215 tablet_service.cc:1467] Processing CreateTablet for tablet 8856f3086b8c49958100db146dc254ed (DEFAULT_TABLE table=test-table [id=2de679ab9f4d4d99a9531de048ab6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:11.252454 18346 tablet_service.cc:1467] Processing CreateTablet for tablet 8856f3086b8c49958100db146dc254ed (DEFAULT_TABLE table=test-table [id=2de679ab9f4d4d99a9531de048ab6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:11.253027 18215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856f3086b8c49958100db146dc254ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:11.253612 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856f3086b8c49958100db146dc254ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:11.253396 18476 tablet_service.cc:1467] Processing CreateTablet for tablet 8856f3086b8c49958100db146dc254ed (DEFAULT_TABLE table=test-table [id=2de679ab9f4d4d99a9531de048ab6440]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:11.254698 18476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8856f3086b8c49958100db146dc254ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:11.264416 18559 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Bootstrap starting.
I20260620 07:02:11.267403 18560 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Bootstrap starting.
I20260620 07:02:11.268096 18559 tablet_bootstrap.cc:654] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:11.268829 18561 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Bootstrap starting.
I20260620 07:02:11.269416 18559 log.cc:824] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:11.270823 18560 tablet_bootstrap.cc:654] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:11.271836 18561 tablet_bootstrap.cc:654] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:11.271994 18560 log.cc:824] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:11.272608 18559 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: No bootstrap required, opened a new log
I20260620 07:02:11.272812 18561 log.cc:824] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:11.273159 18559 ts_tablet_manager.cc:1397] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 07:02:11.275260 18560 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: No bootstrap required, opened a new log
I20260620 07:02:11.275521 18560 ts_tablet_manager.cc:1397] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:02:11.276464 18561 tablet_bootstrap.cc:492] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: No bootstrap required, opened a new log
I20260620 07:02:11.276686 18561 ts_tablet_manager.cc:1397] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 07:02:11.284462 18561 raft_consensus.cc:348] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.284463 18559 raft_consensus.cc:348] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.284910 18559 raft_consensus.cc:374] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:11.285022 18559 raft_consensus.cc:729] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d8a300fa5b494d910fa6ad0df0a04e, State: Initialized, Role: FOLLOWER
I20260620 07:02:11.285075 18561 raft_consensus.cc:374] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:11.285270 18561 raft_consensus.cc:729] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f33e406f666e4733a28c8300d2449a2c, State: Initialized, Role: FOLLOWER
I20260620 07:02:11.285132 18560 raft_consensus.cc:348] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.285760 18560 raft_consensus.cc:374] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:11.285892 18560 raft_consensus.cc:729] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da9927b8153418a9d3ae821cc732460, State: Initialized, Role: FOLLOWER
I20260620 07:02:11.286042 18561 consensus_queue.cc:260] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.285993 18559 consensus_queue.cc:260] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.286649 18560 consensus_queue.cc:260] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.290100 18560 ts_tablet_manager.cc:1428] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 07:02:11.290485 18280 heartbeater.cc:503] Master 127.0.97.62:39897 was elected leader, sending a full tablet report...
I20260620 07:02:11.293365 18411 heartbeater.cc:503] Master 127.0.97.62:39897 was elected leader, sending a full tablet report...
I20260620 07:02:11.293551 18561 ts_tablet_manager.cc:1428] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20260620 07:02:11.293874 18559 ts_tablet_manager.cc:1428] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Time spent starting tablet: real 0.021s user 0.011s sys 0.007s
I20260620 07:02:11.294337 18541 heartbeater.cc:503] Master 127.0.97.62:39897 was elected leader, sending a full tablet report...
I20260620 07:02:11.414516 18565 raft_consensus.cc:482] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:11.414875 18565 raft_consensus.cc:504] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
W20260620 07:02:11.415589 18542 tablet.cc:2369] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:11.416764 18565 leader_election.cc:288] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d8a300fa5b494d910fa6ad0df0a04e (127.0.97.2:34823), 5da9927b8153418a9d3ae821cc732460 (127.0.97.1:33245)
I20260620 07:02:11.423306 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.416968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34823 (local address 127.0.97.3:55737)
0620 07:02:11.417139 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.417158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.417270 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.417746 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.417750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.417781 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:11.418166 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.418180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.419562 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.419568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.421001 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.421013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.422491 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.422497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.422741 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.422775 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.422902 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.423035 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"spinlock_wait_cycles":19712}
I20260620 07:02:11.423503 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.417039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55737 (local address 127.0.97.2:34823)
0620 07:02:11.417216 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.417221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.417242 (+ 21us) server_negotiation.cc:407] Connection header received
0620 07:02:11.417470 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.417475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.417575 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.417666 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:11.418328 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.419387 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.421157 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.422319 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.422465 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.422604 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.422943 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.423156 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:02:11.424783 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.417569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59837 (local address 127.0.97.1:33245)
0620 07:02:11.417764 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.417769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.417786 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:02:11.417947 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.417952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.418034 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.418120 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:11.419555 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.420298 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.422503 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.423700 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.423798 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.423883 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.424406 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.424539 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 07:02:11.424783 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856f3086b8c49958100db146dc254ed" candidate_uuid: "f33e406f666e4733a28c8300d2449a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" is_pre_election: true
I20260620 07:02:11.425369 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.417026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.97.3:59837)
0620 07:02:11.417702 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.417722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:11.417819 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:11.418172 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:11.418177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:11.418220 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:11.418588 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.418600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.420437 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.420442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.422299 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:11.422314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.424022 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.424028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:11.424215 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.424253 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 07:02:11.424963 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 07:02:11.425105 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260620 07:02:11.425335 18366 raft_consensus.cc:2454] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33e406f666e4733a28c8300d2449a2c in term 0.
I20260620 07:02:11.426110 18429 leader_election.cc:302] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7d8a300fa5b494d910fa6ad0df0a04e, f33e406f666e4733a28c8300d2449a2c; no voters:
I20260620 07:02:11.426527 18565 raft_consensus.cc:2781] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:11.426702 18565 raft_consensus.cc:482] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:11.426796 18565 raft_consensus.cc:3037] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:11.426625 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856f3086b8c49958100db146dc254ed" candidate_uuid: "f33e406f666e4733a28c8300d2449a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da9927b8153418a9d3ae821cc732460" is_pre_election: true
I20260620 07:02:11.427181 18235 raft_consensus.cc:2454] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f33e406f666e4733a28c8300d2449a2c in term 0.
I20260620 07:02:11.429490 18565 raft_consensus.cc:504] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.430053 18565 leader_election.cc:288] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [CANDIDATE]: Term 1 election: Requested vote from peers a7d8a300fa5b494d910fa6ad0df0a04e (127.0.97.2:34823), 5da9927b8153418a9d3ae821cc732460 (127.0.97.1:33245)
I20260620 07:02:11.430585 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856f3086b8c49958100db146dc254ed" candidate_uuid: "f33e406f666e4733a28c8300d2449a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da9927b8153418a9d3ae821cc732460"
I20260620 07:02:11.430816 18235 raft_consensus.cc:3037] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:11.433682 18235 raft_consensus.cc:2454] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33e406f666e4733a28c8300d2449a2c in term 1.
I20260620 07:02:11.434134 18432 leader_election.cc:302] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da9927b8153418a9d3ae821cc732460, f33e406f666e4733a28c8300d2449a2c; no voters:
I20260620 07:02:11.434120 18366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8856f3086b8c49958100db146dc254ed" candidate_uuid: "f33e406f666e4733a28c8300d2449a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e"
I20260620 07:02:11.434355 18366 raft_consensus.cc:3037] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:11.434482 18565 raft_consensus.cc:2781] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:11.434957 18565 raft_consensus.cc:686] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 1 LEADER]: Becoming Leader. State: Replica: f33e406f666e4733a28c8300d2449a2c, State: Running, Role: LEADER
I20260620 07:02:11.435578 18565 consensus_queue.cc:237] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } }
I20260620 07:02:11.437427 18366 raft_consensus.cc:2454] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f33e406f666e4733a28c8300d2449a2c in term 1.
I20260620 07:02:11.441226 18094 catalog_manager.cc:5495] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to f33e406f666e4733a28c8300d2449a2c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f33e406f666e4733a28c8300d2449a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f33e406f666e4733a28c8300d2449a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 } health_report { overall_health: UNKNOWN } } }
I20260620 07:02:11.473604 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.466915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.0.97.3:35951)
0620 07:02:11.467090 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.467096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.467158 (+ 62us) server_negotiation.cc:407] Connection header received
0620 07:02:11.467405 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.467412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.467541 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.467698 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:11.468134 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.469245 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.471150 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.471590 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.471708 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.472302 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:11.473016 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:11.473162 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.473280 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.473365 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:02:11.481462 18366 raft_consensus.cc:1261] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 1 FOLLOWER]: Refusing update from remote peer f33e406f666e4733a28c8300d2449a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:11.481485 18235 raft_consensus.cc:1261] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 1 FOLLOWER]: Refusing update from remote peer f33e406f666e4733a28c8300d2449a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:11.482349 18569 consensus_queue.cc:1035] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da9927b8153418a9d3ae821cc732460" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:11.482766 18565 consensus_queue.cc:1035] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d8a300fa5b494d910fa6ad0df0a04e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:02:11.484760 18412 tablet.cc:2369] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:11.494251 18572 mvcc.cc:205] Tried to move back new op lower bound from 7298821863339577344 to 7298821863166103552. Current Snapshot: MvccSnapshot[applied={T|T < 7298821863339577344}]
I20260620 07:02:11.509788 18094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:11.510097 18094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:11.514161 18094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2096f78513054c12a9acc85c127cf257: Sending DeleteTablet for 3 replicas of tablet 8856f3086b8c49958100db146dc254ed
I20260620 07:02:11.515407 18476 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856f3086b8c49958100db146dc254ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:11 UTC) from {username='slave'} at 127.0.0.1:60536
I20260620 07:02:11.515547 18215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856f3086b8c49958100db146dc254ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:11 UTC) from {username='slave'} at 127.0.0.1:47422
I20260620 07:02:11.515543 18346 tablet_service.cc:1514] Processing DeleteTablet for tablet 8856f3086b8c49958100db146dc254ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:11 UTC) from {username='slave'} at 127.0.0.1:49366
I20260620 07:02:11.517021 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18153
I20260620 07:02:11.523092 18584 tablet_replica.cc:330] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: stopping tablet replica
I20260620 07:02:11.523627 18584 raft_consensus.cc:2229] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:11.524207 18584 raft_consensus.cc:2258] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:11.524961 18583 tablet_replica.cc:330] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: stopping tablet replica
I20260620 07:02:11.525411 18583 raft_consensus.cc:2229] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:11.526404 18584 ts_tablet_manager.cc:1905] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:11.526861 18583 raft_consensus.cc:2258] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:11.529166 18583 ts_tablet_manager.cc:1905] T 8856f3086b8c49958100db146dc254ed P 5da9927b8153418a9d3ae821cc732460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:11.532435 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.518430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.97.1:33245)
0620 07:02:11.518660 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.518666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.524070 (+ 5404us) server_negotiation.cc:407] Connection header received
0620 07:02:11.524272 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.524278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.524405 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.524541 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:11.525024 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.526254 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.528315 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.528916 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.529017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.530573 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:11.530589 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:11.530609 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:11.531448 (+ 839us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:11.531514 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:11.531542 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:11.531640 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.531897 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.532035 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 07:02:11.533881 18275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:11.534771 18582 tablet_replica.cc:330] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: stopping tablet replica
I20260620 07:02:11.535233 18582 raft_consensus.cc:2229] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:11.535948 18582 raft_consensus.cc:2258] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:11.537516 18584 ts_tablet_manager.cc:1918] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:11.537719 18584 log.cc:1196] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/8856f3086b8c49958100db146dc254ed
I20260620 07:02:11.538443 18584 ts_tablet_manager.cc:1939] T 8856f3086b8c49958100db146dc254ed P a7d8a300fa5b494d910fa6ad0df0a04e: Deleting consensus metadata
I20260620 07:02:11.539238 18582 ts_tablet_manager.cc:1905] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:11.540576 18078 catalog_manager.cc:4841] TS a7d8a300fa5b494d910fa6ad0df0a04e (127.0.97.2:34823): tablet 8856f3086b8c49958100db146dc254ed (table test-table [id=2de679ab9f4d4d99a9531de048ab6440]) successfully deleted
I20260620 07:02:11.547089 18582 ts_tablet_manager.cc:1918] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:11.547299 18582 log.cc:1196] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/8856f3086b8c49958100db146dc254ed
I20260620 07:02:11.547852 18582 ts_tablet_manager.cc:1939] T 8856f3086b8c49958100db146dc254ed P f33e406f666e4733a28c8300d2449a2c: Deleting consensus metadata
I20260620 07:02:11.549808 18080 catalog_manager.cc:4841] TS f33e406f666e4733a28c8300d2449a2c (127.0.97.3:35951): tablet 8856f3086b8c49958100db146dc254ed (table test-table [id=2de679ab9f4d4d99a9531de048ab6440]) successfully deleted
I20260620 07:02:11.627678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18153
I20260620 07:02:11.649772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18284
W20260620 07:02:11.649852 18081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:33245 (error 108)
I20260620 07:02:11.652370 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.651815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47444)
0620 07:02:11.652070 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.652199 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260620 07:02:11.652758 18081 catalog_manager.cc:4584] TS 5da9927b8153418a9d3ae821cc732460 (127.0.97.1:33245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8856f3086b8c49958100db146dc254ed: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
I20260620 07:02:11.657379 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.651325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.0.97.2:34823)
0620 07:02:11.651568 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.651575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.651595 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:02:11.651680 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.651685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.651789 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.651884 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:11.652351 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.653093 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.654345 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.654798 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.654913 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.656046 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:11.656057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:11.656072 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:11.656652 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:11.656706 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:11.656729 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:11.656824 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.656952 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.657070 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 07:02:11.658046 18406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:11.706714 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.706070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47458)
0620 07:02:11.706421 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.706517 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 07:02:11.709990 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.709488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47468)
0620 07:02:11.709736 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.709839 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 07:02:11.758364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18284
I20260620 07:02:11.770646 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.770164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47480)
0620 07:02:11.770373 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.770463 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 07:02:11.771714 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.771386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47484)
0620 07:02:11.771538 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.771606 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 07:02:11.779332 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18414
I20260620 07:02:11.787705 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.780046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.0.97.3:35951)
0620 07:02:11.780210 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:11.780215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:11.780283 (+ 68us) server_negotiation.cc:407] Connection header received
0620 07:02:11.780511 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:11.780516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:11.780595 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:11.780681 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:11.781190 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.782040 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.783852 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:11.784302 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:11.784417 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:11.785727 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:11.785738 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:11.785757 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:11.786848 (+ 1091us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:11.786933 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:11.786962 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:11.787081 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:11.787216 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 07:02:11.787326 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 07:02:11.788517 18536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:11.852137 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.851646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47486)
0620 07:02:11.851877 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.851966 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 07:02:11.853264 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:11.852873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47494)
0620 07:02:11.853071 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:11.853147 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 07:02:11.907210 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18414
I20260620 07:02:11.922348 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18063
I20260620 07:02:11.923482 18124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:12.030207 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:12.029673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33245 (local address 127.0.0.1:47496)
0620 07:02:12.029886 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:12.030016 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:33245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 07:02:12.030212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18063
2026-06-20T07:02:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2815 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:12 dist-test-slave-mz20 krb5kdc[18593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:12 dist-test-slave-mz20 krb5kdc[18593](info): set up 2 sockets
Jun 20 07:02:12 dist-test-slave-mz20 krb5kdc[18593](info): commencing operation
krb5kdc: starting...
W20260620 07:02:13.054404 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:13 dist-test-slave-mz20 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:13.290280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43179
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:13.621779 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:13.622072 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:13.622206 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:13.629464 18606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:13.629590 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:13.629667 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:13.629739 18606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:13.629796 18606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:13.639607 18606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:13.640937 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:13.642217 18606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:13.653546 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:13.653592 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:13.653788 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:13.654523 18606 server_base.cc:1034] running on GCE node
I20260620 07:02:13.655323 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:13.656731 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:13.657963 18606 hybrid_clock.cc:648] HybridClock initialized: now 1781938933657899 us; error 47 us; skew 500 ppm
I20260620 07:02:13.658319 18606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:13 dist-test-slave-mz20 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:13.661295 18606 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:13.662793 18606 webserver.cc:460] Webserver started at http://127.0.97.62:35403/ using document root <none> and password file <none>
I20260620 07:02:13.663462 18606 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:13.663586 18606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:13.663878 18606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:13.666966 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "c3c1725640a2440085c2f03d7f6c8c5c"
format_stamp: "Formatted at 2026-06-20 07:02:13 on dist-test-slave-mz20"
server_key: "46ba0ee8e461f6e133794c62b426e3d0"
server_key_iv: "08ef076cfeed003338876b2e14049be7"
server_key_version: "encryptionkey@0"
I20260620 07:02:13.667704 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3c1725640a2440085c2f03d7f6c8c5c"
format_stamp: "Formatted at 2026-06-20 07:02:13 on dist-test-slave-mz20"
server_key: "46ba0ee8e461f6e133794c62b426e3d0"
server_key_iv: "08ef076cfeed003338876b2e14049be7"
server_key_version: "encryptionkey@0"
I20260620 07:02:13.672880 18606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:13.676411 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:13.677809 18606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:13.678038 18606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "c3c1725640a2440085c2f03d7f6c8c5c"
format_stamp: "Formatted at 2026-06-20 07:02:13 on dist-test-slave-mz20"
server_key: "46ba0ee8e461f6e133794c62b426e3d0"
server_key_iv: "08ef076cfeed003338876b2e14049be7"
server_key_version: "encryptionkey@0"
I20260620 07:02:13.678236 18606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:13.698918 18606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:13.702755 18606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:13.703014 18606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:13.726787 18606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43179
I20260620 07:02:13.726806 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43179 every 8 connection(s)
I20260620 07:02:13.728207 18606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:13.731778 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:13.733400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18606
I20260620 07:02:13.733592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:13.733963 388 external_mini_cluster.cc:1428] Setting key 6c9024c2ce4bdccb195366489e0cc9fa
I20260620 07:02:13.745246 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: Bootstrap starting.
Jun 20 07:02:13 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:13.748355 18673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:13.749445 18673 log.cc:824] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:13.752247 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: No bootstrap required, opened a new log
I20260620 07:02:13.755860 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:13.735665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.97.62:43179)
0620 07:02:13.736483 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:13.736502 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:13.736560 (+ 58us) server_negotiation.cc:407] Connection header received
0620 07:02:13.738181 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:13.738207 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:13.738738 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:13.739146 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:13.740430 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:13.742390 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:13.743771 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:13.744744 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:13.744926 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:13.749042 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:13.749054 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:13.749071 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:13.749128 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:13.752106 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:13.752771 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:13.752777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:13.752782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:13.752941 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:13.753285 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:13.753289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:13.753292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:13.753796 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:13.754085 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:13.754497 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 07:02:13.754918 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":361,"threads_started":1}
I20260620 07:02:13.759828 18673 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } }
I20260620 07:02:13.760156 18673 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:13.760239 18673 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c1725640a2440085c2f03d7f6c8c5c, State: Initialized, Role: FOLLOWER
I20260620 07:02:13.760768 18673 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } }
I20260620 07:02:13.760996 18673 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:13.761113 18673 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:13.761252 18673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:13.763661 18673 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } }
I20260620 07:02:13.764132 18673 leader_election.cc:302] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c1725640a2440085c2f03d7f6c8c5c; no voters:
I20260620 07:02:13.764670 18673 leader_election.cc:288] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:13.764791 18678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:13.765381 18678 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [term 1 LEADER]: Becoming Leader. State: Replica: c3c1725640a2440085c2f03d7f6c8c5c, State: Running, Role: LEADER
I20260620 07:02:13.765944 18678 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } }
I20260620 07:02:13.766594 18673 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:13.768648 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c1725640a2440085c2f03d7f6c8c5c. Latest consensus state: current_term: 1 leader_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } } }
I20260620 07:02:13.769055 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:13.768994 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1725640a2440085c2f03d7f6c8c5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43179 } } }
I20260620 07:02:13.769758 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:13.771104 18687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:13.776185 18687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:13.785827 18687 catalog_manager.cc:1347] Generated new cluster ID: f01c4c129a00484daf2a1b5d976a5b7e
I20260620 07:02:13.785934 18687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:13.795789 18687 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:13.797067 18687 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:13.803213 18687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: Generated new TSK 0
I20260620 07:02:13.804419 18687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:02:13.877720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:14.235874 18699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:14.236151 18699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:14.236272 18699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:14.243610 18699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:14.243723 18699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:14.243858 18699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:14.253615 18699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:14.254982 18699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:14.256237 18699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:14.268664 18704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:14.268631 18705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:14.268774 18707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:14.269493 18699 server_base.cc:1034] running on GCE node
I20260620 07:02:14.270113 18699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:14.271173 18699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:14.272410 18699 hybrid_clock.cc:648] HybridClock initialized: now 1781938934272363 us; error 38 us; skew 500 ppm
I20260620 07:02:14.272709 18699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:14 dist-test-slave-mz20 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:14.275770 18699 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:02:14.277206 18699 webserver.cc:460] Webserver started at http://127.0.97.1:43093/ using document root <none> and password file <none>
I20260620 07:02:14.277866 18699 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:14.277963 18699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:14.278251 18699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:14.281124 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "a0ef25a43eb0e07496316aaad22ed033"
server_key_iv: "4ea86e7e66436e42883d6547a52a7aa7"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.281877 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "a0ef25a43eb0e07496316aaad22ed033"
server_key_iv: "4ea86e7e66436e42883d6547a52a7aa7"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.286753 18699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:02:14.290051 18714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.291424 18699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:14.291567 18699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "a0ef25a43eb0e07496316aaad22ed033"
server_key_iv: "4ea86e7e66436e42883d6547a52a7aa7"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.291749 18699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:14.305207 18699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:14.309496 18699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:14.309737 18699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:14.310706 18699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:14.311988 18699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:14.312057 18699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.312153 18699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:14.312197 18699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.340540 18699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35011
I20260620 07:02:14.340584 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35011 every 8 connection(s)
I20260620 07:02:14.341794 18699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:14.344424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18699
I20260620 07:02:14.344590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:14.345165 388 external_mini_cluster.cc:1428] Setting key 8ac50f8e149aca5ebc1b4080f804fa19
Jun 20 07:02:14 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:14.364344 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:14.344494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48179 (local address 127.0.97.62:43179)
0620 07:02:14.345019 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:14.345025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:14.346000 (+ 975us) server_negotiation.cc:407] Connection header received
0620 07:02:14.348344 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:14.348348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:14.348415 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:14.348521 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:14.350625 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.351364 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.353472 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.353879 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.353972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:14.358071 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:14.358077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:14.358081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:14.358119 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:14.360772 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:14.361891 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:14.361897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:14.361901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:14.362040 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:14.362583 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:14.362589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:14.362593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:14.362929 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:14.363056 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:14.363897 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 07:02:14.364090 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260620 07:02:14.365648 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:14.345015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43179 (local address 127.0.97.1:48179)
0620 07:02:14.345828 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:14.345871 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:14.348018 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:14.348803 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:14.348815 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:14.349615 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:14.350399 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:14.350425 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.351519 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.351523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:14.353323 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:14.353339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.354090 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.354096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:14.354377 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:14.355089 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:14.355118 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:14.357791 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:14.361013 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:14.361032 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:14.361054 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:14.361679 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:14.362150 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:14.362155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:14.362159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:14.362413 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:14.363049 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:14.363057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:14.363492 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 07:02:14.363788 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 07:02:14.364321 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":304,"threads_started":1}
I20260620 07:02:14.367268 18827 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43179
I20260620 07:02:14.367624 18827 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:14.368482 18827 heartbeater.cc:511] Master 127.0.97.62:43179 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 07:02:14.370728 18638 ts_manager.cc:194] Registered new tserver with Master: 0a3d7f12ecc84f40ac7e8e1deb8705c2 (127.0.97.1:35011)
I20260620 07:02:14.372262 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:02:14.399469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:14.714877 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:14.715157 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:14.715277 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:14.723763 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:14.723884 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:14.724042 18833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:14.734380 18833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:14.735714 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:14.736994 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:14.749975 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:14.750001 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:14.749923 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:14.750793 18833 server_base.cc:1034] running on GCE node
I20260620 07:02:14.751379 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:14.752339 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:14.753523 18833 hybrid_clock.cc:648] HybridClock initialized: now 1781938934753471 us; error 54 us; skew 500 ppm
I20260620 07:02:14.753850 18833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:14 dist-test-slave-mz20 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:14.757083 18833 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:02:14.758666 18833 webserver.cc:460] Webserver started at http://127.0.97.2:46413/ using document root <none> and password file <none>
I20260620 07:02:14.759336 18833 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:14.759449 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:14.759739 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:14.762739 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "b63a3ba17fef4a1f9e4cec958f5615ea"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "8bb890653a72e49de7db6dff8c2adaf4"
server_key_iv: "136085d76aa997b7000e3b58fc64889f"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.763576 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b63a3ba17fef4a1f9e4cec958f5615ea"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "8bb890653a72e49de7db6dff8c2adaf4"
server_key_iv: "136085d76aa997b7000e3b58fc64889f"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.768523 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:14.771872 18848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.773334 18833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 07:02:14.773483 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "b63a3ba17fef4a1f9e4cec958f5615ea"
format_stamp: "Formatted at 2026-06-20 07:02:14 on dist-test-slave-mz20"
server_key: "8bb890653a72e49de7db6dff8c2adaf4"
server_key_iv: "136085d76aa997b7000e3b58fc64889f"
server_key_version: "encryptionkey@0"
I20260620 07:02:14.773684 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:14.788048 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:14.792353 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:14.792624 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:14.793607 18833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:14.795002 18833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:14.795078 18833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.795192 18833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:14.795246 18833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:14.826089 18833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45785
I20260620 07:02:14.826133 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45785 every 8 connection(s)
I20260620 07:02:14.827364 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:14.831743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18833
I20260620 07:02:14.831884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:14.832276 388 external_mini_cluster.cc:1428] Setting key a192ba4f1058ceb7cdf147d5a600f0de
Jun 20 07:02:14 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:14.848067 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:14.829958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41417 (local address 127.0.97.62:43179)
0620 07:02:14.830136 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:14.830141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:14.831158 (+ 1017us) server_negotiation.cc:407] Connection header received
0620 07:02:14.833048 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:14.833054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:14.833139 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:14.833247 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:14.835417 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.836453 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.838002 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.838583 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.838701 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:14.842334 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:14.842339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:14.842342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:14.842376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:14.844900 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:14.845731 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:14.845736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:14.845738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:14.845873 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:14.846379 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:14.846383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:14.846385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:14.846714 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:14.846831 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:14.847623 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 07:02:14.847812 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:02:14.849386 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:14.830308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43179 (local address 127.0.97.2:41417)
0620 07:02:14.831007 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:14.831042 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:14.832767 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:14.834131 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:14.834143 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:14.834722 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:14.835244 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:14.835260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.836620 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.836624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:14.837869 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:14.837882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:14.838817 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:14.838823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:14.839036 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:14.839706 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:14.839729 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:14.842063 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:14.845057 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:14.845065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:14.845083 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:14.845610 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:14.845966 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:14.845970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:14.845973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:14.846229 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:14.846824 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:14.846831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:14.847254 (+ 423us) client_negotiation.cc:769] Sending connection context
0620 07:02:14.847535 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 07:02:14.848068 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":289,"threads_started":1}
I20260620 07:02:14.851089 18961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43179
I20260620 07:02:14.851408 18961 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:14.852097 18961 heartbeater.cc:511] Master 127.0.97.62:43179 requested a full tablet report, sending...
I20260620 07:02:14.853608 18638 ts_manager.cc:194] Registered new tserver with Master: b63a3ba17fef4a1f9e4cec958f5615ea (127.0.97.2:45785)
I20260620 07:02:14.854759 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41417
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:02:14.885365 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:15.207968 18966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:15.208241 18966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:15.208334 18966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:15.215759 18966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:15.215888 18966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:15.216012 18966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:15.226003 18966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:15.227368 18966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:15.228657 18966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:15.241180 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:15.241211 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:15.241255 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:15.242375 18966 server_base.cc:1034] running on GCE node
I20260620 07:02:15.243041 18966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:15.243978 18966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:15.245179 18966 hybrid_clock.cc:648] HybridClock initialized: now 1781938935245138 us; error 28 us; skew 500 ppm
I20260620 07:02:15.245508 18966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:15 dist-test-slave-mz20 krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:15.248553 18966 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:02:15.249905 18966 webserver.cc:460] Webserver started at http://127.0.97.3:34635/ using document root <none> and password file <none>
I20260620 07:02:15.250561 18966 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:15.250671 18966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:15.250942 18966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:15.253738 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "160ee6edff5b4c74b3d87b1553128715"
format_stamp: "Formatted at 2026-06-20 07:02:15 on dist-test-slave-mz20"
server_key: "f340da53d3332a3562bab755a514acc5"
server_key_iv: "1501f6364f30ccc94225c52f02142bec"
server_key_version: "encryptionkey@0"
I20260620 07:02:15.254484 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "160ee6edff5b4c74b3d87b1553128715"
format_stamp: "Formatted at 2026-06-20 07:02:15 on dist-test-slave-mz20"
server_key: "f340da53d3332a3562bab755a514acc5"
server_key_iv: "1501f6364f30ccc94225c52f02142bec"
server_key_version: "encryptionkey@0"
I20260620 07:02:15.259578 18966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:02:15.262926 18981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:15.264235 18966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:02:15.264386 18966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "160ee6edff5b4c74b3d87b1553128715"
format_stamp: "Formatted at 2026-06-20 07:02:15 on dist-test-slave-mz20"
server_key: "f340da53d3332a3562bab755a514acc5"
server_key_iv: "1501f6364f30ccc94225c52f02142bec"
server_key_version: "encryptionkey@0"
I20260620 07:02:15.264565 18966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:15.294207 18966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:15.298558 18966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:15.298831 18966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:15.299775 18966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:15.301106 18966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:15.301192 18966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:15.301312 18966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:15.301373 18966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:15.332494 18966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34245
I20260620 07:02:15.332540 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34245 every 8 connection(s)
I20260620 07:02:15.333699 18966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:15.338526 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 18966
I20260620 07:02:15.338743 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:15.339242 388 external_mini_cluster.cc:1428] Setting key d96af079f919001f48909d7f8f3e86ef
Jun 20 07:02:15 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:15.357084 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.336199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50969 (local address 127.0.97.62:43179)
0620 07:02:15.336376 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.336382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.337584 (+ 1202us) server_negotiation.cc:407] Connection header received
0620 07:02:15.339894 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.339900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.339992 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.340121 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:15.342072 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.342894 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.345235 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.345689 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.345796 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.350256 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:15.350266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:15.350271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:15.350334 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:15.353828 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.354757 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.354763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.354766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.354903 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.355372 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.355377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.355381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.355656 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:15.355787 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.356524 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.356755 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:02:15.358055 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.336642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43179 (local address 127.0.97.3:50969)
0620 07:02:15.337402 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.337442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.339565 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.340431 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.340443 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.341171 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:15.341914 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.341938 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.343033 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.343039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.344996 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.345012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.345908 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.345915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.346205 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.347022 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:15.347053 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:15.349866 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:15.353973 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:15.353984 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:15.354006 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:15.354651 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:15.355008 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:15.355013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:15.355018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:15.355256 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:15.355787 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:15.355795 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:15.356148 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.356385 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.356926 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":299,"threads_started":1}
I20260620 07:02:15.359352 19094 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43179
I20260620 07:02:15.359611 19094 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:15.360260 19094 heartbeater.cc:511] Master 127.0.97.62:43179 requested a full tablet report, sending...
I20260620 07:02:15.361749 18638 ts_manager.cc:194] Registered new tserver with Master: 160ee6edff5b4c74b3d87b1553128715 (127.0.97.3:34245)
I20260620 07:02:15.362572 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50969
I20260620 07:02:15.367116 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:15.375734 18827 heartbeater.cc:503] Master 127.0.97.62:43179 was elected leader, sending a full tablet report...
I20260620 07:02:15.383489 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.371070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.0.97.62:43179)
0620 07:02:15.371253 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.371260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.371619 (+ 359us) server_negotiation.cc:407] Connection header received
0620 07:02:15.371771 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.371776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.371838 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.371916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:15.373275 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.374023 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.375300 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.375687 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.375792 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.378420 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:15.378424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:15.378428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:15.378458 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:15.380734 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.381519 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.381523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.381525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.381638 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.382053 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.382057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.382059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.382398 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:15.382670 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.383029 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.383206 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 07:02:15.387128 18638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:15.394001 18638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:15.418295 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.406935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.97.2:45785)
0620 07:02:15.408951 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.408960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.408991 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:02:15.409107 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.409116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.409380 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.410373 (+ 993us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:15.410648 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.412355 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.415368 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.417039 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.417185 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.417393 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.417777 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.417965 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1751,"threads_started":1}
I20260620 07:02:15.418278 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.406759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45785 (local address 127.0.0.1:36044)
0620 07:02:15.407703 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.407743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.407917 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.409638 (+ 1721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.409647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.409683 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:15.410155 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.410167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.412529 (+ 2362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.412535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.415103 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.415122 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.417279 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.417286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.417493 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.417546 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.417702 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.417905 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":938,"threads_started":1}
I20260620 07:02:15.422925 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.407055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34245 (local address 127.0.0.1:45716)
0620 07:02:15.414892 (+ 7837us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.414912 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.415548 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.416246 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.416251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.416275 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:15.416607 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.416617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.418505 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.418509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.420483 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.420499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.422134 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.422141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.422302 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.422336 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.422423 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.422581 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7706,"thread_start_us":7538,"threads_started":1}
I20260620 07:02:15.422967 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.407451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.97.3:34245)
0620 07:02:15.408545 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.408553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.415028 (+ 6475us) server_negotiation.cc:407] Connection header received
0620 07:02:15.415778 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.415789 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.416067 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.416256 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:15.416842 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.418338 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.420668 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.421945 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.422061 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.422287 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.422485 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.422660 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":806,"threads_started":1}
I20260620 07:02:15.424037 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.408792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35011 (local address 127.0.0.1:45060)
0620 07:02:15.415766 (+ 6974us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.415785 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.415901 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.416811 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.416815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.416842 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:15.417207 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.417221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.418971 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.418976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.421444 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.421459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.423330 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.423336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.423494 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.423526 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.423604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.423727 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6833,"thread_start_us":6873,"threads_started":1}
I20260620 07:02:15.424331 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.409279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.0.97.1:35011)
0620 07:02:15.415284 (+ 6005us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.415294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.415884 (+ 590us) server_negotiation.cc:407] Connection header received
0620 07:02:15.416060 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.416068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.416274 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.416458 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:15.417406 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.418837 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.421621 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.422998 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.423140 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.423351 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.423676 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.423839 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5787,"thread_start_us":5940,"threads_started":1}
I20260620 07:02:15.423110 18896 tablet_service.cc:1467] Processing CreateTablet for tablet 8de2cf11961643608b3626b7a247245e (DEFAULT_TABLE table=test-table [id=2451b4e0027948f29d9725306e5e13af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:15.425417 18896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de2cf11961643608b3626b7a247245e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:15.426996 19029 tablet_service.cc:1467] Processing CreateTablet for tablet 8de2cf11961643608b3626b7a247245e (DEFAULT_TABLE table=test-table [id=2451b4e0027948f29d9725306e5e13af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:15.427920 18762 tablet_service.cc:1467] Processing CreateTablet for tablet 8de2cf11961643608b3626b7a247245e (DEFAULT_TABLE table=test-table [id=2451b4e0027948f29d9725306e5e13af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:15.428447 19029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de2cf11961643608b3626b7a247245e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:15.429184 18762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8de2cf11961643608b3626b7a247245e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:15.439509 19112 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Bootstrap starting.
I20260620 07:02:15.441591 19113 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Bootstrap starting.
I20260620 07:02:15.443338 19112 tablet_bootstrap.cc:654] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:15.444602 19112 log.cc:824] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:15.444701 19114 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Bootstrap starting.
I20260620 07:02:15.445508 19113 tablet_bootstrap.cc:654] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:15.446836 19113 log.cc:824] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:15.448356 19114 tablet_bootstrap.cc:654] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:15.448412 19112 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: No bootstrap required, opened a new log
I20260620 07:02:15.448660 19112 ts_tablet_manager.cc:1397] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260620 07:02:15.449545 19114 log.cc:824] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:15.450490 19113 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: No bootstrap required, opened a new log
I20260620 07:02:15.450809 19113 ts_tablet_manager.cc:1397] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260620 07:02:15.455828 19112 raft_consensus.cc:348] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.456243 19112 raft_consensus.cc:374] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:15.456338 19112 raft_consensus.cc:729] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63a3ba17fef4a1f9e4cec958f5615ea, State: Initialized, Role: FOLLOWER
I20260620 07:02:15.456947 19112 consensus_queue.cc:260] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.458106 18961 heartbeater.cc:503] Master 127.0.97.62:43179 was elected leader, sending a full tablet report...
I20260620 07:02:15.458843 19112 ts_tablet_manager.cc:1428] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Time spent starting tablet: real 0.010s user 0.009s sys 0.003s
I20260620 07:02:15.459123 19114 tablet_bootstrap.cc:492] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: No bootstrap required, opened a new log
I20260620 07:02:15.459424 19114 ts_tablet_manager.cc:1397] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260620 07:02:15.463376 19113 raft_consensus.cc:348] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.463855 19113 raft_consensus.cc:374] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:15.463958 19113 raft_consensus.cc:729] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 160ee6edff5b4c74b3d87b1553128715, State: Initialized, Role: FOLLOWER
I20260620 07:02:15.464617 19113 consensus_queue.cc:260] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.465783 19094 heartbeater.cc:503] Master 127.0.97.62:43179 was elected leader, sending a full tablet report...
I20260620 07:02:15.466531 19113 ts_tablet_manager.cc:1428] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260620 07:02:15.469823 19114 raft_consensus.cc:348] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.470330 19114 raft_consensus.cc:374] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:15.470433 19114 raft_consensus.cc:729] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a3d7f12ecc84f40ac7e8e1deb8705c2, State: Initialized, Role: FOLLOWER
I20260620 07:02:15.471138 19114 consensus_queue.cc:260] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.473623 19114 ts_tablet_manager.cc:1428] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260620 07:02:15.535656 19120 raft_consensus.cc:482] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:15.535948 19120 raft_consensus.cc:504] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.537621 19120 leader_election.cc:288] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b63a3ba17fef4a1f9e4cec958f5615ea (127.0.97.2:45785), 160ee6edff5b4c74b3d87b1553128715 (127.0.97.3:34245)
I20260620 07:02:15.544184 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.538089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41131 (local address 127.0.97.3:34245)
0620 07:02:15.538296 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.538302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.539009 (+ 707us) server_negotiation.cc:407] Connection header received
0620 07:02:15.539195 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.539200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.539261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.539334 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:15.539913 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.540610 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.542481 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.543306 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.543419 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.543522 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.543823 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.543946 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 07:02:15.544296 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.538072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55585 (local address 127.0.97.2:45785)
0620 07:02:15.538306 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.538312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.539190 (+ 878us) server_negotiation.cc:407] Connection header received
0620 07:02:15.539341 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.539345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.539404 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.539477 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:15.540039 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.540726 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.542595 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.543410 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.543533 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.543630 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.543924 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.544054 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:02:15.544343 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.538097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34245 (local address 127.0.97.1:41131)
0620 07:02:15.538885 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.538902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.539059 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.539367 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.539371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.539396 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:15.539764 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.539774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.540739 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.540746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.542324 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.542339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.543511 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.543517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.543662 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.543694 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.543778 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.543910 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":55,"thread_start_us":471,"threads_started":1}
I20260620 07:02:15.544648 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.538142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45785 (local address 127.0.97.1:55585)
0620 07:02:15.539072 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:15.539087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:15.539218 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:15.539523 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:15.539528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:15.539553 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:15.539883 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.539892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.540866 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.540870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.542368 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:15.542379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.543641 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.543648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:15.543783 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.543806 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:02:15.543884 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 07:02:15.544006 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":22,"thread_start_us":605,"threads_started":1}
I20260620 07:02:15.545375 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de2cf11961643608b3626b7a247245e" candidate_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "160ee6edff5b4c74b3d87b1553128715" is_pre_election: true
I20260620 07:02:15.545544 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de2cf11961643608b3626b7a247245e" candidate_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" is_pre_election: true
I20260620 07:02:15.546030 18916 raft_consensus.cc:2454] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3d7f12ecc84f40ac7e8e1deb8705c2 in term 0.
I20260620 07:02:15.546017 19049 raft_consensus.cc:2454] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a3d7f12ecc84f40ac7e8e1deb8705c2 in term 0.
I20260620 07:02:15.546725 18716 leader_election.cc:302] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3d7f12ecc84f40ac7e8e1deb8705c2, b63a3ba17fef4a1f9e4cec958f5615ea; no voters:
I20260620 07:02:15.547109 19120 raft_consensus.cc:2781] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:15.547235 19120 raft_consensus.cc:482] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:15.547323 19120 raft_consensus.cc:3037] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:15.549481 19120 raft_consensus.cc:504] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.550067 19120 leader_election.cc:288] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [CANDIDATE]: Term 1 election: Requested vote from peers b63a3ba17fef4a1f9e4cec958f5615ea (127.0.97.2:45785), 160ee6edff5b4c74b3d87b1553128715 (127.0.97.3:34245)
I20260620 07:02:15.550561 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de2cf11961643608b3626b7a247245e" candidate_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea"
I20260620 07:02:15.550587 19049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8de2cf11961643608b3626b7a247245e" candidate_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "160ee6edff5b4c74b3d87b1553128715"
I20260620 07:02:15.550776 19049 raft_consensus.cc:3037] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:15.550782 18916 raft_consensus.cc:3037] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:15.552884 18916 raft_consensus.cc:2454] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3d7f12ecc84f40ac7e8e1deb8705c2 in term 1.
I20260620 07:02:15.552906 19049 raft_consensus.cc:2454] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a3d7f12ecc84f40ac7e8e1deb8705c2 in term 1.
I20260620 07:02:15.553269 18718 leader_election.cc:302] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a3d7f12ecc84f40ac7e8e1deb8705c2, 160ee6edff5b4c74b3d87b1553128715; no voters:
I20260620 07:02:15.553552 19120 raft_consensus.cc:2781] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:15.553951 19120 raft_consensus.cc:686] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a3d7f12ecc84f40ac7e8e1deb8705c2, State: Running, Role: LEADER
I20260620 07:02:15.554440 19120 consensus_queue.cc:237] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } }
I20260620 07:02:15.558712 18638 catalog_manager.cc:5495] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a3d7f12ecc84f40ac7e8e1deb8705c2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a3d7f12ecc84f40ac7e8e1deb8705c2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35011 } health_report { overall_health: HEALTHY } } }
I20260620 07:02:15.574827 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.569373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.97.1:35011)
0620 07:02:15.569572 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.569578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.569605 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:02:15.569782 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.569786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.569879 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.570037 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:15.570439 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.571208 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.572638 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.573042 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.573147 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.573680 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:15.574177 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:15.574314 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.574463 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.574504 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 07:02:15.579288 18962 tablet.cc:2369] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:15.582960 18916 raft_consensus.cc:1261] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 1 FOLLOWER]: Refusing update from remote peer 0a3d7f12ecc84f40ac7e8e1deb8705c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:15.582954 19049 raft_consensus.cc:1261] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 1 FOLLOWER]: Refusing update from remote peer 0a3d7f12ecc84f40ac7e8e1deb8705c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:15.583956 19123 consensus_queue.cc:1035] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b63a3ba17fef4a1f9e4cec958f5615ea" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:15.584362 19120 consensus_queue.cc:1035] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "160ee6edff5b4c74b3d87b1553128715" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 07:02:15.585249 19095 tablet.cc:2369] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:02:15.594277 18828 tablet.cc:2369] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:15.595038 19126 mvcc.cc:205] Tried to move back new op lower bound from 7298821880137961472 to 7298821880035147776. Current Snapshot: MvccSnapshot[applied={T|T < 7298821880137961472}]
I20260620 07:02:15.608160 18638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:15.608414 18638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:15.611987 18638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3c1725640a2440085c2f03d7f6c8c5c: Sending DeleteTablet for 3 replicas of tablet 8de2cf11961643608b3626b7a247245e
I20260620 07:02:15.613284 19029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de2cf11961643608b3626b7a247245e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45716
I20260620 07:02:15.613281 18896 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de2cf11961643608b3626b7a247245e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36044
I20260620 07:02:15.613596 18762 tablet_service.cc:1514] Processing DeleteTablet for tablet 8de2cf11961643608b3626b7a247245e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45060
I20260620 07:02:15.614390 19137 tablet_replica.cc:330] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: stopping tablet replica
I20260620 07:02:15.614629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18699
I20260620 07:02:15.614858 19137 raft_consensus.cc:2229] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:15.615495 19137 raft_consensus.cc:2258] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:15.617553 19136 tablet_replica.cc:330] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: stopping tablet replica
I20260620 07:02:15.618122 19136 raft_consensus.cc:2229] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:15.618362 19137 ts_tablet_manager.cc:1905] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:15.618806 19136 raft_consensus.cc:2258] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:15.620850 19136 ts_tablet_manager.cc:1905] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:15.625922 19137 ts_tablet_manager.cc:1918] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:15.626138 19137 log.cc:1196] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/8de2cf11961643608b3626b7a247245e
I20260620 07:02:15.626686 19137 ts_tablet_manager.cc:1939] T 8de2cf11961643608b3626b7a247245e P b63a3ba17fef4a1f9e4cec958f5615ea: Deleting consensus metadata
I20260620 07:02:15.627707 19136 ts_tablet_manager.cc:1918] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:15.627887 19136 log.cc:1196] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/8de2cf11961643608b3626b7a247245e
I20260620 07:02:15.628405 19136 ts_tablet_manager.cc:1939] T 8de2cf11961643608b3626b7a247245e P 160ee6edff5b4c74b3d87b1553128715: Deleting consensus metadata
I20260620 07:02:15.628754 18623 catalog_manager.cc:4841] TS b63a3ba17fef4a1f9e4cec958f5615ea (127.0.97.2:45785): tablet 8de2cf11961643608b3626b7a247245e (table test-table [id=2451b4e0027948f29d9725306e5e13af]) successfully deleted
I20260620 07:02:15.629796 19138 tablet_replica.cc:330] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: stopping tablet replica
I20260620 07:02:15.630210 18625 catalog_manager.cc:4841] TS 160ee6edff5b4c74b3d87b1553128715 (127.0.97.3:34245): tablet 8de2cf11961643608b3626b7a247245e (table test-table [id=2451b4e0027948f29d9725306e5e13af]) successfully deleted
I20260620 07:02:15.630769 19138 raft_consensus.cc:2229] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:15.631616 19138 raft_consensus.cc:2258] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:15.633898 19138 ts_tablet_manager.cc:1905] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:15.639935 19138 ts_tablet_manager.cc:1918] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:15.640130 19138 log.cc:1196] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/8de2cf11961643608b3626b7a247245e
I20260620 07:02:15.640635 19138 ts_tablet_manager.cc:1939] T 8de2cf11961643608b3626b7a247245e P 0a3d7f12ecc84f40ac7e8e1deb8705c2: Deleting consensus metadata
I20260620 07:02:15.642263 18623 catalog_manager.cc:4841] TS 0a3d7f12ecc84f40ac7e8e1deb8705c2 (127.0.97.1:35011): tablet 8de2cf11961643608b3626b7a247245e (table test-table [id=2451b4e0027948f29d9725306e5e13af]) successfully deleted
Jun 20 07:02:15 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:02:15.653914 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.615585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45074 (local address 127.0.97.1:35011)
0620 07:02:15.615771 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.615778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.632370 (+ 16592us) server_negotiation.cc:407] Connection header received
0620 07:02:15.632580 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.632587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.632668 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.632760 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:15.638146 (+ 5386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.639461 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.641256 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.641954 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.642092 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.646418 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:15.646437 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:15.646458 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:15.646508 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:15.650622 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.651470 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.651481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.651485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.651696 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.652192 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.652197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.652201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.652694 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:15.652984 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.653271 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.653535 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:02:15.654778 18822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:15.755607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18699
I20260620 07:02:15.771533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18833
Jun 20 07:02:15 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:02:15.787211 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.772359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.97.2:45785)
0620 07:02:15.772567 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.772573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.772593 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:02:15.772660 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.772666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.772744 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.772823 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:15.774021 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.774844 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.776096 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.776484 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.776591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.781042 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:15.781057 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:15.781074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:15.781149 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:15.784122 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.784868 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.784878 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.784880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.785016 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.785422 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.785428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.785431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.785821 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:15.786128 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.786647 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.786880 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:02:15.787935 18956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:15.877763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18833
I20260620 07:02:15.891772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18966
Jun 20 07:02:15 dist-test-slave-mz20 krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:02:15.906762 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:15.892564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.97.3:34245)
0620 07:02:15.892742 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:15.892747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:15.892765 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:02:15.892889 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:15.892893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:15.892970 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:15.893052 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:15.894187 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.894994 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.896240 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:15.896670 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:15.896776 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:15.900906 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:15.900925 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:15.900943 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:15.900982 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:15.903669 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.904474 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.904483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.904485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.904664 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:15.905062 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:15.905065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:15.905067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:15.905443 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:15.905693 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:15.906011 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 07:02:15.906214 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:02:15.907487 19089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:15.996023 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18966
I20260620 07:02:16.009222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 18606
I20260620 07:02:16.010324 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:16.119668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 18606
2026-06-20T07:02:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4090 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T07:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:16Z Disabled control of system clock
I20260620 07:02:16.190042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38715
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:16.517282 19147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:16.517586 19147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:16.517712 19147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:16.525048 19147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:16.525163 19147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:16.525205 19147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:16.525285 19147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:16.525342 19147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:16.534672 19147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:16.535899 19147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:16.537667 19147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:16.549201 19152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:16.549135 19153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:16.549185 19155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:16.549795 19147 server_base.cc:1034] running on GCE node
I20260620 07:02:16.550617 19147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:16.552050 19147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:16.553258 19147 hybrid_clock.cc:648] HybridClock initialized: now 1781938936553211 us; error 44 us; skew 500 ppm
I20260620 07:02:16.553644 19147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:16.556033 19147 webserver.cc:460] Webserver started at http://127.0.97.62:40297/ using document root <none> and password file <none>
I20260620 07:02:16.556723 19147 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:16.556816 19147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:16.557085 19147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:16.560050 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "0681e89600c84b29aa2d7c76f79090f7"
format_stamp: "Formatted at 2026-06-20 07:02:16 on dist-test-slave-mz20"
server_key: "5079f66711e69c15264cbeadf27c0780"
server_key_iv: "6718ee8ceb6d7d1d39b4572246739cf3"
server_key_version: "encryptionkey@0"
I20260620 07:02:16.560842 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "0681e89600c84b29aa2d7c76f79090f7"
format_stamp: "Formatted at 2026-06-20 07:02:16 on dist-test-slave-mz20"
server_key: "5079f66711e69c15264cbeadf27c0780"
server_key_iv: "6718ee8ceb6d7d1d39b4572246739cf3"
server_key_version: "encryptionkey@0"
I20260620 07:02:16.565999 19147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:02:16.569530 19161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:16.570932 19147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:16.571096 19147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "0681e89600c84b29aa2d7c76f79090f7"
format_stamp: "Formatted at 2026-06-20 07:02:16 on dist-test-slave-mz20"
server_key: "5079f66711e69c15264cbeadf27c0780"
server_key_iv: "6718ee8ceb6d7d1d39b4572246739cf3"
server_key_version: "encryptionkey@0"
I20260620 07:02:16.571296 19147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:16.583866 19147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:16.584692 19147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:16.584966 19147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:16.602948 19147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38715
I20260620 07:02:16.602953 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38715 every 8 connection(s)
I20260620 07:02:16.604311 19147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:16.608060 19213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:16.613703 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19147
I20260620 07:02:16.613870 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:16.614290 388 external_mini_cluster.cc:1428] Setting key 7a53dc4d3bccb63f0c669487d8562daa
I20260620 07:02:16.619493 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: Bootstrap starting.
I20260620 07:02:16.622862 19213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:16.624060 19213 log.cc:824] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:16.627213 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: No bootstrap required, opened a new log
I20260620 07:02:16.628218 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:16.616369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.97.62:38715)
0620 07:02:16.617427 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:16.617439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:16.617483 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:02:16.619506 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:16.619529 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:16.620016 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:16.620326 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:16.620912 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:16.622175 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:16.623996 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:16.624833 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:16.625022 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:16.626042 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:16.626049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:16.626062 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:16.626602 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:16.626677 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:16.626701 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:16.626783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:16.626986 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 07:02:16.627292 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":386,"threads_started":1}
I20260620 07:02:16.635138 19213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } }
I20260620 07:02:16.635505 19213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:16.635596 19213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0681e89600c84b29aa2d7c76f79090f7, State: Initialized, Role: FOLLOWER
I20260620 07:02:16.636133 19213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } }
I20260620 07:02:16.636369 19213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:16.636492 19213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:16.636650 19213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:16.639108 19213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } }
I20260620 07:02:16.639734 19213 leader_election.cc:302] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0681e89600c84b29aa2d7c76f79090f7; no voters:
I20260620 07:02:16.640396 19213 leader_election.cc:288] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:16.640525 19218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:16.641132 19218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [term 1 LEADER]: Becoming Leader. State: Replica: 0681e89600c84b29aa2d7c76f79090f7, State: Running, Role: LEADER
I20260620 07:02:16.641654 19218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } }
I20260620 07:02:16.642721 19213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:16.644416 19219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0681e89600c84b29aa2d7c76f79090f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } } }
I20260620 07:02:16.644820 19219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:16.646437 19220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0681e89600c84b29aa2d7c76f79090f7. Latest consensus state: current_term: 1 leader_uuid: "0681e89600c84b29aa2d7c76f79090f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0681e89600c84b29aa2d7c76f79090f7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38715 } } }
I20260620 07:02:16.646641 19223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:16.646718 19220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:16.652698 19223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:16.661360 19223 catalog_manager.cc:1347] Generated new cluster ID: 41fa01b631ae4f3da076e0fd09e75fb0
I20260620 07:02:16.661572 19223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:16.673763 19223 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:16.675657 19223 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:16.688973 19223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: Generated new TSK 0
I20260620 07:02:16.689879 19223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:16.697170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--builtin_ntp_servers=127.0.97.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:17.023077 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:17.023371 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:17.023483 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:17.030881 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:17.030993 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:17.031126 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:17.040485 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:17.041811 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:17.043735 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:17.056849 19243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.056849 19245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.056871 19242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:17.057411 19237 server_base.cc:1034] running on GCE node
I20260620 07:02:17.058060 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:17.059072 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:17.060279 19237 hybrid_clock.cc:648] HybridClock initialized: now 1781938937060217 us; error 51 us; skew 500 ppm
I20260620 07:02:17.060614 19237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:17.062893 19237 webserver.cc:460] Webserver started at http://127.0.97.1:38203/ using document root <none> and password file <none>
I20260620 07:02:17.063563 19237 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:17.063659 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:17.063956 19237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:17.066797 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "d74ad337f6ea4d65a538dd2f4327ce49"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "b2b38a3201e246d076d88c8610e78a2a"
server_key_iv: "8715dae31f5ba210169496406fd045d6"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.067546 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d74ad337f6ea4d65a538dd2f4327ce49"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "b2b38a3201e246d076d88c8610e78a2a"
server_key_iv: "8715dae31f5ba210169496406fd045d6"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.072592 19237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:17.076131 19251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.077491 19237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:02:17.077637 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "d74ad337f6ea4d65a538dd2f4327ce49"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "b2b38a3201e246d076d88c8610e78a2a"
server_key_iv: "8715dae31f5ba210169496406fd045d6"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.077842 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:17.092950 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:17.093781 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:17.094043 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:17.095077 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:17.096565 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:17.096661 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.096771 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:17.096839 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.127013 19237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37525
I20260620 07:02:17.127089 19363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37525 every 8 connection(s)
I20260620 07:02:17.128283 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:17.132715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19237
I20260620 07:02:17.132860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:17.133255 388 external_mini_cluster.cc:1428] Setting key 9899a0182bc86cfa5cf2a6ac3acda000
I20260620 07:02:17.136039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--builtin_ntp_servers=127.0.97.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:17.144184 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:17.130709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46015 (local address 127.0.97.62:38715)
0620 07:02:17.131118 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:17.131123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:17.132324 (+ 1201us) server_negotiation.cc:407] Connection header received
0620 07:02:17.134236 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:17.134240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:17.134302 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:17.134392 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:17.135608 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.136598 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.138145 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.138693 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.138807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:17.143339 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:17.143344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:17.143348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:17.143367 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:17.143413 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:17.143427 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:17.143501 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:17.143724 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 07:02:17.143830 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260620 07:02:17.145473 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:17.131244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38715 (local address 127.0.97.1:46015)
0620 07:02:17.132154 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:17.132195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:17.133999 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:17.134654 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:17.134665 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:17.134784 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:17.135445 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:17.135462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.136764 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.136769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:17.138007 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:17.138020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.138911 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.138916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:17.139122 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:17.142977 (+ 3855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:17.142980 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:17.143036 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:17.143135 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:17.143517 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:17.143529 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:17.143536 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:02:17.143683 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 07:02:17.144127 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":302,"threads_started":1}
I20260620 07:02:17.147197 19364 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38715
I20260620 07:02:17.147495 19364 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:17.148248 19364 heartbeater.cc:511] Master 127.0.97.62:38715 requested a full tablet report, sending...
I20260620 07:02:17.150429 19178 ts_manager.cc:194] Registered new tserver with Master: d74ad337f6ea4d65a538dd2f4327ce49 (127.0.97.1:37525)
I20260620 07:02:17.152416 19178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46015
W20260620 07:02:17.456211 19368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:17.456489 19368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:17.456583 19368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:17.463788 19368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:17.463896 19368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:17.464002 19368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:17.476356 19368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:17.477574 19368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:17.479390 19368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:17.491909 19374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.491866 19376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.491945 19373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:17.492681 19368 server_base.cc:1034] running on GCE node
I20260620 07:02:17.493261 19368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:17.494158 19368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:17.495357 19368 hybrid_clock.cc:648] HybridClock initialized: now 1781938937495307 us; error 43 us; skew 500 ppm
I20260620 07:02:17.495646 19368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:17.497754 19368 webserver.cc:460] Webserver started at http://127.0.97.2:33519/ using document root <none> and password file <none>
I20260620 07:02:17.498436 19368 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:17.498559 19368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:17.498867 19368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:17.501597 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "b168a5cfd2fb499399dfbbca5813dda2"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "9249bb534c756cedbf3980e00120d7ff"
server_key_iv: "2cc4c19c78a745d8fb576669011f84be"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.502354 19368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b168a5cfd2fb499399dfbbca5813dda2"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "9249bb534c756cedbf3980e00120d7ff"
server_key_iv: "2cc4c19c78a745d8fb576669011f84be"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.507323 19368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 07:02:17.510527 19382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.511822 19368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:02:17.511969 19368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "b168a5cfd2fb499399dfbbca5813dda2"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "9249bb534c756cedbf3980e00120d7ff"
server_key_iv: "2cc4c19c78a745d8fb576669011f84be"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.512167 19368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:17.524365 19368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:17.525233 19368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:17.525490 19368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:17.526454 19368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:17.527988 19368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:17.528085 19368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.528196 19368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:17.528275 19368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.558079 19368 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36215
I20260620 07:02:17.558128 19494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36215 every 8 connection(s)
I20260620 07:02:17.559397 19368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:17.560931 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19368
I20260620 07:02:17.561086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:17.561671 388 external_mini_cluster.cc:1428] Setting key b8639179665f46c79513aaca2b0afdd5
I20260620 07:02:17.564874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--builtin_ntp_servers=127.0.97.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:17.573585 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:17.561899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35947 (local address 127.0.97.62:38715)
0620 07:02:17.562051 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:17.562056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:17.563568 (+ 1512us) server_negotiation.cc:407] Connection header received
0620 07:02:17.566042 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:17.566047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:17.566118 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:17.566215 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:17.567603 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.568366 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.570527 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.571171 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.571272 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:17.572785 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:17.572789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:17.572793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:17.572812 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:17.572845 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:17.572858 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:17.572928 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:17.573164 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:02:17.573264 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:02:17.574605 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:17.562524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38715 (local address 127.0.97.2:35947)
0620 07:02:17.563423 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:17.563477 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:17.565695 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:17.566451 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:17.566463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:17.566618 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:17.567395 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:17.567419 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.568517 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.568521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:17.570366 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:17.570381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:17.571396 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:17.571402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:17.571668 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:17.572398 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:17.572401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:17.572472 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:17.572571 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:17.572942 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:17.572951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:17.572958 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 07:02:17.573138 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 07:02:17.573484 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":735,"threads_started":1}
I20260620 07:02:17.576033 19495 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38715
I20260620 07:02:17.576380 19495 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:17.577247 19495 heartbeater.cc:511] Master 127.0.97.62:38715 requested a full tablet report, sending...
I20260620 07:02:17.578787 19178 ts_manager.cc:194] Registered new tserver with Master: b168a5cfd2fb499399dfbbca5813dda2 (127.0.97.2:36215)
I20260620 07:02:17.579595 19178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35947
W20260620 07:02:17.889551 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:17.889843 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:17.889948 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:17.897292 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:17.897405 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:17.897541 19498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:17.907375 19498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38715
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:17.908711 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:17.910503 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:17.923861 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.923820 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:17.923902 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:17.924880 19498 server_base.cc:1034] running on GCE node
I20260620 07:02:17.925464 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:17.926358 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:17.927570 19498 hybrid_clock.cc:648] HybridClock initialized: now 1781938937927502 us; error 60 us; skew 500 ppm
I20260620 07:02:17.927902 19498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:17.930042 19498 webserver.cc:460] Webserver started at http://127.0.97.3:34241/ using document root <none> and password file <none>
I20260620 07:02:17.930712 19498 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:17.930809 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:17.931100 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:17.933792 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "f014db75012d4da6b4cd68c5c1c13589"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "1ccea0ef42b3189d181f8ed2566af941"
server_key_iv: "8c6a5d24a0f809cc521eaa9a2308591b"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.934523 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f014db75012d4da6b4cd68c5c1c13589"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "1ccea0ef42b3189d181f8ed2566af941"
server_key_iv: "8c6a5d24a0f809cc521eaa9a2308591b"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.939465 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:02:17.943034 19512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.944384 19498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:02:17.944545 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "f014db75012d4da6b4cd68c5c1c13589"
format_stamp: "Formatted at 2026-06-20 07:02:17 on dist-test-slave-mz20"
server_key: "1ccea0ef42b3189d181f8ed2566af941"
server_key_iv: "8c6a5d24a0f809cc521eaa9a2308591b"
server_key_version: "encryptionkey@0"
I20260620 07:02:17.944751 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:17.963625 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:17.964452 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:17.964707 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:17.965691 19498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:17.967203 19498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:17.967301 19498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.967417 19498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:17.967486 19498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:17.997200 19498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37119
I20260620 07:02:17.997246 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37119 every 8 connection(s)
I20260620 07:02:17.998448 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:18.001925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19498
I20260620 07:02:18.002097 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:18.002517 388 external_mini_cluster.cc:1428] Setting key 36e48ac5689932b73235a4f87c40d36b
I20260620 07:02:18.010783 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.000925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47915 (local address 127.0.97.62:38715)
0620 07:02:18.001122 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.001127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.002795 (+ 1668us) server_negotiation.cc:407] Connection header received
0620 07:02:18.004558 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.004564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.004649 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.004735 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:18.005831 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.006576 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.008130 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.008543 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.008652 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.009939 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:18.009945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:18.009948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:18.009966 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:18.009999 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:18.010012 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:18.010104 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.010315 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.010435 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:02:18.011739 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.001409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38715 (local address 127.0.97.3:47915)
0620 07:02:18.002597 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.002643 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.004287 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.005002 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.005011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.005105 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:18.005658 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.005675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.006730 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.006734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.007984 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.007997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.008757 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.008764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.008981 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.009631 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:18.009634 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:18.009672 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:18.009757 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:18.010095 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:18.010107 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:18.010115 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.010287 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.010717 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":416,"threads_started":1}
I20260620 07:02:18.013049 19625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38715
I20260620 07:02:18.013326 19625 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:18.014096 19625 heartbeater.cc:511] Master 127.0.97.62:38715 requested a full tablet report, sending...
I20260620 07:02:18.015569 19178 ts_manager.cc:194] Registered new tserver with Master: f014db75012d4da6b4cd68c5c1c13589 (127.0.97.3:37119)
I20260620 07:02:18.016615 19178 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47915
I20260620 07:02:18.017560 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:18.027633 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.021741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.0.97.62:38715)
0620 07:02:18.021960 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.021968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.022288 (+ 320us) server_negotiation.cc:407] Connection header received
0620 07:02:18.022446 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.022450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.022525 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.022660 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:18.023234 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.023971 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.025352 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.025743 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.025847 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.026892 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:18.026896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:18.026900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:18.026919 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:18.026949 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:18.026961 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:18.027099 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.027285 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.027331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:02:18.031733 19177 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:18.038499 19177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:18.076591 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.051115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37119 (local address 127.0.0.1:45842)
0620 07:02:18.066157 (+ 15042us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.066225 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.066481 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.069009 (+ 2528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.069016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.069060 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:18.069503 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.069520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.071308 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.071313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.073265 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.073277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.075433 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.075439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.075842 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.075882 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.076015 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.076215 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14711,"mutex_wait_us":96,"thread_start_us":15621,"threads_started":1}
I20260620 07:02:18.076586 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.051176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.97.3:37119)
0620 07:02:18.068330 (+ 17154us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.068346 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.068377 (+ 31us) server_negotiation.cc:407] Connection header received
0620 07:02:18.068477 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.068484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.068757 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.068976 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:18.069701 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.071155 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.073473 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.074722 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.074840 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.075717 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.076072 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.076239 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16972,"thread_start_us":16827,"threads_started":1}
I20260620 07:02:18.076880 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.052221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.97.1:37525)
0620 07:02:18.060385 (+ 8164us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.060397 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.068001 (+ 7604us) server_negotiation.cc:407] Connection header received
0620 07:02:18.068682 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.068688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.068850 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.069022 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:18.069816 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.071453 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.073997 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.075496 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.075616 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.075821 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.076359 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.076526 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7957,"thread_start_us":7831,"threads_started":1}
I20260620 07:02:18.077628 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.051955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37525 (local address 127.0.0.1:55634)
0620 07:02:18.067865 (+ 15910us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.067881 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.068522 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.069252 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.069256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.069283 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:18.069655 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.069666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.071772 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.071777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.073788 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.073801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.075961 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.075967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.076152 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.076185 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.077127 (+ 942us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.077267 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15797,"mutex_wait_us":2,"thread_start_us":15025,"threads_started":1}
I20260620 07:02:18.077919 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.051164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36215 (local address 127.0.0.1:46534)
0620 07:02:18.067280 (+ 16116us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.067298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.067429 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.070834 (+ 3405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.070838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.070862 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:18.071270 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.071283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.073150 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.073156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.075072 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.075086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.077182 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.077188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.077353 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.077384 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.077466 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.077593 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15979,"thread_start_us":16522,"threads_started":1}
I20260620 07:02:18.080479 19560 tablet_service.cc:1467] Processing CreateTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (DEFAULT_TABLE table=test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:18.081095 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.051630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.0.97.2:36215)
0620 07:02:18.070069 (+ 18439us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.070077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.070103 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:02:18.070194 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.070201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.070417 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.070786 (+ 369us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:18.071468 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.073002 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.075313 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.076923 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.077128 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.077350 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.077548 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.077720 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18273,"thread_start_us":18912,"threads_started":1}
I20260620 07:02:18.080806 19299 tablet_service.cc:1467] Processing CreateTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (DEFAULT_TABLE table=test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:18.082079 19560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:18.082099 19299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:18.084918 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (DEFAULT_TABLE table=test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:18.086287 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:18.095981 19643 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Bootstrap starting.
I20260620 07:02:18.099215 19644 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Bootstrap starting.
I20260620 07:02:18.099272 19643 tablet_bootstrap.cc:654] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:18.100708 19643 log.cc:824] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:18.101017 19645 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Bootstrap starting.
I20260620 07:02:18.102834 19644 tablet_bootstrap.cc:654] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:18.104115 19644 log.cc:824] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:18.104619 19645 tablet_bootstrap.cc:654] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:18.104779 19643 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: No bootstrap required, opened a new log
I20260620 07:02:18.105093 19643 ts_tablet_manager.cc:1397] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 07:02:18.105893 19645 log.cc:824] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:18.107007 19644 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: No bootstrap required, opened a new log
I20260620 07:02:18.107283 19644 ts_tablet_manager.cc:1397] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:02:18.108932 19645 tablet_bootstrap.cc:492] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: No bootstrap required, opened a new log
I20260620 07:02:18.109193 19645 ts_tablet_manager.cc:1397] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 07:02:18.114506 19644 raft_consensus.cc:348] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.114912 19644 raft_consensus.cc:374] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:18.115010 19644 raft_consensus.cc:729] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f014db75012d4da6b4cd68c5c1c13589, State: Initialized, Role: FOLLOWER
I20260620 07:02:18.115146 19643 raft_consensus.cc:348] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.115582 19643 raft_consensus.cc:374] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:18.115744 19643 raft_consensus.cc:729] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74ad337f6ea4d65a538dd2f4327ce49, State: Initialized, Role: FOLLOWER
I20260620 07:02:18.115600 19644 consensus_queue.cc:260] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.116313 19643 consensus_queue.cc:260] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.117694 19364 heartbeater.cc:503] Master 127.0.97.62:38715 was elected leader, sending a full tablet report...
I20260620 07:02:18.118322 19643 ts_tablet_manager.cc:1428] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 07:02:18.119048 19645 raft_consensus.cc:348] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.119503 19645 raft_consensus.cc:374] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:18.119606 19645 raft_consensus.cc:729] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b168a5cfd2fb499399dfbbca5813dda2, State: Initialized, Role: FOLLOWER
I20260620 07:02:18.120301 19625 heartbeater.cc:503] Master 127.0.97.62:38715 was elected leader, sending a full tablet report...
I20260620 07:02:18.120263 19645 consensus_queue.cc:260] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.121073 19644 ts_tablet_manager.cc:1428] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 07:02:18.121867 19495 heartbeater.cc:503] Master 127.0.97.62:38715 was elected leader, sending a full tablet report...
I20260620 07:02:18.122152 19649 raft_consensus.cc:482] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:18.122504 19645 ts_tablet_manager.cc:1428] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 07:02:18.122435 19649 raft_consensus.cc:504] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.128293 19649 leader_election.cc:288] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b168a5cfd2fb499399dfbbca5813dda2 (127.0.97.2:36215), d74ad337f6ea4d65a538dd2f4327ce49 (127.0.97.1:37525)
W20260620 07:02:18.130532 19365 tablet.cc:2369] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:18.135393 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.124925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37525 (local address 127.0.97.3:58463)
0620 07:02:18.125361 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.125381 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.125497 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.128048 (+ 2551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.128054 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.128086 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:18.129199 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.129213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.130641 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.130646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.132660 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.132676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.134638 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.134644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.134832 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.134865 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.134934 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.135077 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260620 07:02:18.135907 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.125440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58463 (local address 127.0.97.1:37525)
0620 07:02:18.125603 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.125608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.125626 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:02:18.125692 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.125697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.125779 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.125869 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:18.129376 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.130446 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.133124 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.134361 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.134479 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.135486 (+ 1007us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.135537 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.135664 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:02:18.136925 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659f8ae6d5e4d958f62f48e7c7a61cf" candidate_uuid: "f014db75012d4da6b4cd68c5c1c13589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" is_pre_election: true
I20260620 07:02:18.137480 19319 raft_consensus.cc:2454] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f014db75012d4da6b4cd68c5c1c13589 in term 0.
I20260620 07:02:18.138250 19515 leader_election.cc:302] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d74ad337f6ea4d65a538dd2f4327ce49, f014db75012d4da6b4cd68c5c1c13589; no voters:
I20260620 07:02:18.138700 19651 raft_consensus.cc:2781] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:18.138859 19651 raft_consensus.cc:482] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:18.138988 19651 raft_consensus.cc:3037] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:18.140290 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.125178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34557 (local address 127.0.97.2:36215)
0620 07:02:18.125354 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.125360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.134499 (+ 9139us) server_negotiation.cc:407] Connection header received
0620 07:02:18.134776 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.134780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.134860 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.134935 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:18.135431 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.136398 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.138259 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.139415 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.139540 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.139627 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.139929 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.140053 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:02:18.140636 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.124934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36215 (local address 127.0.97.3:34557)
0620 07:02:18.134395 (+ 9461us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:18.134414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:18.134633 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:18.134974 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:18.134979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:18.135004 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:18.135280 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.135288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.136567 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.136572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.138083 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:18.138098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.139579 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.139585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:18.139761 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.139794 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 07:02:18.139891 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 07:02:18.140022 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9331,"mutex_wait_us":560,"thread_start_us":8635,"threads_started":1}
I20260620 07:02:18.141709 19450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659f8ae6d5e4d958f62f48e7c7a61cf" candidate_uuid: "f014db75012d4da6b4cd68c5c1c13589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b168a5cfd2fb499399dfbbca5813dda2" is_pre_election: true
I20260620 07:02:18.141839 19651 raft_consensus.cc:504] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.142256 19450 raft_consensus.cc:2454] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f014db75012d4da6b4cd68c5c1c13589 in term 0.
I20260620 07:02:18.142701 19651 leader_election.cc:288] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [CANDIDATE]: Term 1 election: Requested vote from peers b168a5cfd2fb499399dfbbca5813dda2 (127.0.97.2:36215), d74ad337f6ea4d65a538dd2f4327ce49 (127.0.97.1:37525)
I20260620 07:02:18.143195 19319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659f8ae6d5e4d958f62f48e7c7a61cf" candidate_uuid: "f014db75012d4da6b4cd68c5c1c13589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74ad337f6ea4d65a538dd2f4327ce49"
I20260620 07:02:18.143430 19319 raft_consensus.cc:3037] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:18.145594 19319 raft_consensus.cc:2454] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f014db75012d4da6b4cd68c5c1c13589 in term 1.
I20260620 07:02:18.145938 19450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7659f8ae6d5e4d958f62f48e7c7a61cf" candidate_uuid: "f014db75012d4da6b4cd68c5c1c13589" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b168a5cfd2fb499399dfbbca5813dda2"
I20260620 07:02:18.146008 19515 leader_election.cc:302] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d74ad337f6ea4d65a538dd2f4327ce49, f014db75012d4da6b4cd68c5c1c13589; no voters:
I20260620 07:02:18.146158 19450 raft_consensus.cc:3037] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:18.146306 19651 raft_consensus.cc:2781] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:18.146564 19651 raft_consensus.cc:686] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 1 LEADER]: Becoming Leader. State: Replica: f014db75012d4da6b4cd68c5c1c13589, State: Running, Role: LEADER
I20260620 07:02:18.147037 19651 consensus_queue.cc:237] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } }
I20260620 07:02:18.148937 19450 raft_consensus.cc:2454] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f014db75012d4da6b4cd68c5c1c13589 in term 1.
I20260620 07:02:18.151563 19176 catalog_manager.cc:5495] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 reported cstate change: term changed from 0 to 1, leader changed from <none> to f014db75012d4da6b4cd68c5c1c13589 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "f014db75012d4da6b4cd68c5c1c13589" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f014db75012d4da6b4cd68c5c1c13589" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 } health_report { overall_health: UNKNOWN } } }
I20260620 07:02:18.183552 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.178638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.0.97.3:37119)
0620 07:02:18.178813 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.178817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.178834 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:02:18.178937 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.178940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.179021 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.179134 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:18.179443 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.180130 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.181456 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.181852 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.181942 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.182482 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:18.182994 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:18.183115 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.183256 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.183290 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:02:18.191197 19319 raft_consensus.cc:1261] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 1 FOLLOWER]: Refusing update from remote peer f014db75012d4da6b4cd68c5c1c13589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:18.191207 19450 raft_consensus.cc:1261] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 1 FOLLOWER]: Refusing update from remote peer f014db75012d4da6b4cd68c5c1c13589: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:18.192044 19651 consensus_queue.cc:1035] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74ad337f6ea4d65a538dd2f4327ce49" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:18.192289 19649 consensus_queue.cc:1035] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b168a5cfd2fb499399dfbbca5813dda2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:18.202261 19656 mvcc.cc:205] Tried to move back new op lower bound from 7298821890821902336 to 7298821890654695424. Current Snapshot: MvccSnapshot[applied={T|T < 7298821890821902336}]
I20260620 07:02:18.215445 19176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:18.215731 19176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:18.219784 19176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0681e89600c84b29aa2d7c76f79090f7: Sending DeleteTablet for 3 replicas of tablet 7659f8ae6d5e4d958f62f48e7c7a61cf
I20260620 07:02:18.220924 19560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:18 UTC) from {username='slave'} at 127.0.0.1:45842
I20260620 07:02:18.220937 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:18 UTC) from {username='slave'} at 127.0.0.1:46534
I20260620 07:02:18.221009 19299 tablet_service.cc:1514] Processing DeleteTablet for tablet 7659f8ae6d5e4d958f62f48e7c7a61cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:18 UTC) from {username='slave'} at 127.0.0.1:55634
I20260620 07:02:18.225207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19237
I20260620 07:02:18.234062 19669 tablet_replica.cc:330] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: stopping tablet replica
I20260620 07:02:18.234732 19669 raft_consensus.cc:2229] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:18.234802 19667 tablet_replica.cc:330] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: stopping tablet replica
I20260620 07:02:18.235221 19667 raft_consensus.cc:2229] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:18.235646 19669 raft_consensus.cc:2258] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:18.235783 19667 raft_consensus.cc:2258] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:18.237046 19668 tablet_replica.cc:330] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: stopping tablet replica
I20260620 07:02:18.237447 19668 raft_consensus.cc:2229] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:18.237964 19668 raft_consensus.cc:2258] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:18.237962 19669 ts_tablet_manager.cc:1905] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:18.238267 19667 ts_tablet_manager.cc:1905] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:18.239936 19668 ts_tablet_manager.cc:1905] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:18.245697 19669 ts_tablet_manager.cc:1918] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:18.245994 19669 log.cc:1196] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/7659f8ae6d5e4d958f62f48e7c7a61cf
I20260620 07:02:18.246753 19669 ts_tablet_manager.cc:1939] T 7659f8ae6d5e4d958f62f48e7c7a61cf P f014db75012d4da6b4cd68c5c1c13589: Deleting consensus metadata
I20260620 07:02:18.246951 19668 ts_tablet_manager.cc:1918] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:18.247130 19668 log.cc:1196] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/7659f8ae6d5e4d958f62f48e7c7a61cf
I20260620 07:02:18.247648 19668 ts_tablet_manager.cc:1939] T 7659f8ae6d5e4d958f62f48e7c7a61cf P b168a5cfd2fb499399dfbbca5813dda2: Deleting consensus metadata
I20260620 07:02:18.249004 19163 catalog_manager.cc:4841] TS f014db75012d4da6b4cd68c5c1c13589 (127.0.97.3:37119): tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (table test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]) successfully deleted
I20260620 07:02:18.249826 19162 catalog_manager.cc:4841] TS b168a5cfd2fb499399dfbbca5813dda2 (127.0.97.2:36215): tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (table test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]) successfully deleted
I20260620 07:02:18.253611 19667 ts_tablet_manager.cc:1918] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:18.253863 19667 log.cc:1196] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/7659f8ae6d5e4d958f62f48e7c7a61cf
I20260620 07:02:18.254159 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.227076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.97.1:37525)
0620 07:02:18.227294 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.227300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.246683 (+ 19383us) server_negotiation.cc:407] Connection header received
0620 07:02:18.247230 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.247236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.247371 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.247454 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:18.248271 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.249457 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.251028 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.251465 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.251582 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.252609 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:18.252620 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:18.252639 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:18.253332 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:18.253379 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:18.253402 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:18.253505 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.253641 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.253761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:02:18.254520 19667 ts_tablet_manager.cc:1939] T 7659f8ae6d5e4d958f62f48e7c7a61cf P d74ad337f6ea4d65a538dd2f4327ce49: Deleting consensus metadata
I20260620 07:02:18.254863 19359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:18.344766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19237
I20260620 07:02:18.344794 19164 catalog_manager.cc:4841] TS d74ad337f6ea4d65a538dd2f4327ce49 (127.0.97.1:37525): tablet 7659f8ae6d5e4d958f62f48e7c7a61cf (table test-table [id=40c79cbe57a04ad6bf41ea184a6ee95d]) successfully deleted
I20260620 07:02:18.360167 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19368
I20260620 07:02:18.367827 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.360956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.97.2:36215)
0620 07:02:18.361151 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.361158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.361186 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:02:18.361350 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.361357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.361472 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.361591 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:18.361939 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.363238 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.364475 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.365115 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.365239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.366167 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:18.366178 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:18.366198 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:18.366986 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:18.367047 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:18.367078 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:18.367192 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.367354 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.367464 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:02:18.368520 19490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:18.454909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19368
I20260620 07:02:18.476150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19498
I20260620 07:02:18.483819 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:18.476932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.97.3:37119)
0620 07:02:18.477144 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:18.477150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:18.477177 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:02:18.477267 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:18.477273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:18.477368 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:18.477467 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:18.477878 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.479079 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.480431 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:18.481060 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:18.481192 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:18.482247 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:18.482258 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:18.482277 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:18.483086 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:18.483132 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:18.483152 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:18.483242 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:18.483402 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 07:02:18.483480 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:02:18.484575 19620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:18.602921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19498
I20260620 07:02:18.621222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19147
I20260620 07:02:18.622221 19208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:18.729947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19147
2026-06-20T07:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2609 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:18 dist-test-slave-mz20 krb5kdc[19677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:18 dist-test-slave-mz20 krb5kdc[19677](info): set up 2 sockets
Jun 20 07:02:18 dist-test-slave-mz20 krb5kdc[19677](info): commencing operation
krb5kdc: starting...
W20260620 07:02:19.746598 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:19 dist-test-slave-mz20 krb5kdc[19677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:19.981556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43765
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:20.308450 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:20.308753 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:20.308884 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:20.316231 19690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:20.316344 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:20.316401 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:20.316484 19690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:20.316540 19690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:20.326696 19690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:20.328022 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:20.329320 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:20.341126 19695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:20.341055 19696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:20.341101 19698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:20.341933 19690 server_base.cc:1034] running on GCE node
I20260620 07:02:20.342725 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:20.344168 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:20.345386 19690 hybrid_clock.cc:648] HybridClock initialized: now 1781938940345334 us; error 57 us; skew 500 ppm
I20260620 07:02:20.345702 19690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:20 dist-test-slave-mz20 krb5kdc[19677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:20.348695 19690 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:20.350222 19690 webserver.cc:460] Webserver started at http://127.0.97.62:42937/ using document root <none> and password file <none>
I20260620 07:02:20.350922 19690 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:20.351033 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:20.351305 19690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:20.354233 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "5d258a4d89984d5daf0118217b8c14f4"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "bf5ab2fac4b4c062afaa22f116a8cf27"
server_key_iv: "fd4796df3fc410ba5c772aff14f3afbc"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.355031 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "5d258a4d89984d5daf0118217b8c14f4"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "bf5ab2fac4b4c062afaa22f116a8cf27"
server_key_iv: "fd4796df3fc410ba5c772aff14f3afbc"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.360729 19690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 07:02:20.364562 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:20.365826 19690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:20.365963 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "5d258a4d89984d5daf0118217b8c14f4"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "bf5ab2fac4b4c062afaa22f116a8cf27"
server_key_iv: "fd4796df3fc410ba5c772aff14f3afbc"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.366125 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:20.407522 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:20.411468 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:20.411697 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:20.429289 19690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43765
I20260620 07:02:20.429293 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43765 every 8 connection(s)
I20260620 07:02:20.430622 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:20.434273 19757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:20.435009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19690
I20260620 07:02:20.435181 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:20.435619 388 external_mini_cluster.cc:1428] Setting key 957098d0ee9eea48858008db3c82e50d
I20260620 07:02:20.448002 19757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: Bootstrap starting.
Jun 20 07:02:20 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:20.451166 19757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:20.452498 19757 log.cc:824] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:20.455891 19757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: No bootstrap required, opened a new log
I20260620 07:02:20.458655 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:20.437416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.97.62:43765)
0620 07:02:20.438164 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:20.438179 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:20.438226 (+ 47us) server_negotiation.cc:407] Connection header received
0620 07:02:20.440484 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:20.440513 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:20.441040 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:20.441472 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:20.442754 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:20.444641 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:20.445987 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:20.447038 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:20.447213 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:20.451205 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:20.451217 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:20.451235 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:20.451290 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:20.454825 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:20.455497 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:20.455505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:20.455512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:20.455728 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:20.456064 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:20.456070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:20.456073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:20.456590 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:20.456801 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:20.457264 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 07:02:20.457580 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260620 07:02:20.463640 19757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } }
I20260620 07:02:20.463989 19757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:20.464073 19757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d258a4d89984d5daf0118217b8c14f4, State: Initialized, Role: FOLLOWER
I20260620 07:02:20.464576 19757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } }
I20260620 07:02:20.464811 19757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:20.464933 19757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:20.465075 19757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:20.467262 19757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } }
I20260620 07:02:20.467727 19757 leader_election.cc:302] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d258a4d89984d5daf0118217b8c14f4; no voters:
I20260620 07:02:20.468236 19757 leader_election.cc:288] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:20.468361 19762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:20.469012 19762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [term 1 LEADER]: Becoming Leader. State: Replica: 5d258a4d89984d5daf0118217b8c14f4, State: Running, Role: LEADER
I20260620 07:02:20.469580 19762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } }
I20260620 07:02:20.470115 19757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:20.472226 19764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d258a4d89984d5daf0118217b8c14f4. Latest consensus state: current_term: 1 leader_uuid: "5d258a4d89984d5daf0118217b8c14f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } } }
I20260620 07:02:20.472196 19763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d258a4d89984d5daf0118217b8c14f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d258a4d89984d5daf0118217b8c14f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43765 } } }
I20260620 07:02:20.472638 19764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:20.472738 19763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:20.474642 19767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:20.479200 19767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:20.487671 19767 catalog_manager.cc:1347] Generated new cluster ID: 5ffc61bef6bc45b390e53b33146d2c4f
I20260620 07:02:20.487790 19767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:20.500204 19767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:20.502039 19767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:20.519616 19767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: Generated new TSK 0
I20260620 07:02:20.520507 19767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 07:02:20.577831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:20.900998 19783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:20.901293 19783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:20.901428 19783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:20.908892 19783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:20.909015 19783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:20.909166 19783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:20.919106 19783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:20.920385 19783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:20.921643 19783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:20.934312 19788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:20.934407 19791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:20.934407 19789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:20.935199 19783 server_base.cc:1034] running on GCE node
I20260620 07:02:20.935770 19783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:20.936695 19783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:20.937913 19783 hybrid_clock.cc:648] HybridClock initialized: now 1781938940937871 us; error 28 us; skew 500 ppm
I20260620 07:02:20.938221 19783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:20 dist-test-slave-mz20 krb5kdc[19677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:20.941150 19783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 07:02:20.942634 19783 webserver.cc:460] Webserver started at http://127.0.97.1:41501/ using document root <none> and password file <none>
I20260620 07:02:20.943265 19783 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:20.943396 19783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:20.943686 19783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:20.946452 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "5a6756646aee4665a732a4f92687d3df"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "2e55dc9e70436fe02c275fbff09ea09a"
server_key_iv: "22b04ae0d5364bce367f02c721363824"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.947225 19783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5a6756646aee4665a732a4f92687d3df"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "2e55dc9e70436fe02c275fbff09ea09a"
server_key_iv: "22b04ae0d5364bce367f02c721363824"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.952018 19783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 07:02:20.955253 19798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:20.956509 19783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:20.956653 19783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "5a6756646aee4665a732a4f92687d3df"
format_stamp: "Formatted at 2026-06-20 07:02:20 on dist-test-slave-mz20"
server_key: "2e55dc9e70436fe02c275fbff09ea09a"
server_key_iv: "22b04ae0d5364bce367f02c721363824"
server_key_version: "encryptionkey@0"
I20260620 07:02:20.956825 19783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:20.969273 19783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:20.973511 19783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:20.973773 19783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:20.974795 19783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:20.976101 19783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:20.976179 19783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:20.976291 19783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:20.976347 19783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.003688 19783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36349
I20260620 07:02:21.003741 19910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36349 every 8 connection(s)
I20260620 07:02:21.004848 19783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:21.010416 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19783
I20260620 07:02:21.010623 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:21.011168 388 external_mini_cluster.cc:1428] Setting key 047ff6b45a6945ca060d7595dab48ab0
Jun 20 07:02:21 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:21.024675 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.007201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49103 (local address 127.0.97.62:43765)
0620 07:02:21.007622 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:21.007627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:21.008442 (+ 815us) server_negotiation.cc:407] Connection header received
0620 07:02:21.010284 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:21.010289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:21.010372 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:21.010467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:21.012364 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.013111 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.014601 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.014994 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.015109 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:21.018587 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:21.018593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:21.018596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:21.018629 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:21.021471 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:21.022228 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:21.022234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:21.022238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:21.022373 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:21.022902 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:21.022907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:21.022910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:21.023252 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:21.023408 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:21.024125 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 07:02:21.024346 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":217,"threads_started":1}
I20260620 07:02:21.025544 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.007618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43765 (local address 127.0.97.1:49103)
0620 07:02:21.008277 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:21.008311 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:21.009968 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:21.010761 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:21.010773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:21.011454 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:21.012199 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:21.012225 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.013247 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.013252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:21.014439 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:21.014450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.015211 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.015218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:21.015432 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:21.016045 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:21.016066 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:21.018321 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:21.021653 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:21.021660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:21.021673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:21.022112 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:21.022486 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:21.022490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:21.022492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:21.022754 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:21.023438 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:21.023444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:21.023778 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 07:02:21.024021 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 07:02:21.024430 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":259,"threads_started":1}
I20260620 07:02:21.027041 19911 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43765
I20260620 07:02:21.027352 19911 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:21.028083 19911 heartbeater.cc:511] Master 127.0.97.62:43765 requested a full tablet report, sending...
I20260620 07:02:21.030112 19722 ts_manager.cc:194] Registered new tserver with Master: 5a6756646aee4665a732a4f92687d3df (127.0.97.1:36349)
I20260620 07:02:21.031634 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49103
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 07:02:21.061574 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:21.372689 19917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:21.372956 19917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:21.373072 19917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:21.380393 19917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:21.380499 19917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:21.380638 19917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:21.390441 19917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:21.391737 19917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:21.392998 19917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:21.405380 19923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:21.405349 19922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:21.405503 19925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:21.406307 19917 server_base.cc:1034] running on GCE node
I20260620 07:02:21.406939 19917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:21.407837 19917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:21.409066 19917 hybrid_clock.cc:648] HybridClock initialized: now 1781938941409010 us; error 39 us; skew 500 ppm
I20260620 07:02:21.409433 19917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:21 dist-test-slave-mz20 krb5kdc[19677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:21.412580 19917 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 07:02:21.414036 19917 webserver.cc:460] Webserver started at http://127.0.97.2:34085/ using document root <none> and password file <none>
I20260620 07:02:21.414781 19917 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:21.414902 19917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:21.415216 19917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:21.418175 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "bfd01b07edd1479696b6d92e53d079f8"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "71655f5633bf2f3319c4d802f740ef29"
server_key_iv: "7d8ddd9288cceec0b3d21f316b05663a"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.419059 19917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bfd01b07edd1479696b6d92e53d079f8"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "71655f5633bf2f3319c4d802f740ef29"
server_key_iv: "7d8ddd9288cceec0b3d21f316b05663a"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.425017 19917 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260620 07:02:21.428733 19932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.430372 19917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:21.430585 19917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "bfd01b07edd1479696b6d92e53d079f8"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "71655f5633bf2f3319c4d802f740ef29"
server_key_iv: "7d8ddd9288cceec0b3d21f316b05663a"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.430835 19917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:21.452198 19917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:21.457655 19917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:21.457954 19917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:21.459110 19917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:21.460791 19917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:21.460877 19917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.460999 19917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:21.461081 19917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.504633 19917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37247
I20260620 07:02:21.504679 20044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37247 every 8 connection(s)
I20260620 07:02:21.505750 19917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:21.514407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 19917
I20260620 07:02:21.514609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:21.515153 388 external_mini_cluster.cc:1428] Setting key 5b4f757c1995051933eef228dd6ac503
Jun 20 07:02:21 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:21.526098 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.508278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41049 (local address 127.0.97.62:43765)
0620 07:02:21.508493 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:21.508498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:21.509562 (+ 1064us) server_negotiation.cc:407] Connection header received
0620 07:02:21.511392 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:21.511396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:21.511467 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:21.511547 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:21.512936 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.513630 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.515773 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.516308 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.516417 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:21.519860 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:21.519866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:21.519870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:21.519908 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:21.522892 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:21.523810 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:21.523815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:21.523819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:21.523993 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:21.524429 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:21.524433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:21.524437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:21.524777 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:21.524922 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:21.525673 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 07:02:21.525860 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 07:02:21.527524 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.508701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43765 (local address 127.0.97.2:41049)
0620 07:02:21.509402 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:21.509446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:21.511129 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:21.511747 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:21.511755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:21.512281 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:21.512800 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:21.512815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.513773 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.513777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:21.515615 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:21.515630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:21.516520 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:21.516527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:21.516736 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:21.517331 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:21.517353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:21.519607 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:21.523099 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:21.523107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:21.523121 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:21.523612 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:21.524107 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:21.524111 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:21.524113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:21.524327 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:21.524902 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:21.524908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:21.525271 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 07:02:21.525586 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 07:02:21.526130 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":263,"threads_started":1}
I20260620 07:02:21.528910 20045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43765
I20260620 07:02:21.529188 20045 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:21.529886 20045 heartbeater.cc:511] Master 127.0.97.62:43765 requested a full tablet report, sending...
I20260620 07:02:21.531360 19722 ts_manager.cc:194] Registered new tserver with Master: bfd01b07edd1479696b6d92e53d079f8 (127.0.97.2:37247)
I20260620 07:02:21.532166 19722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41049
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 07:02:21.570196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:21.890847 20050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:21.891145 20050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:21.891288 20050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:21.898469 20050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:21.898629 20050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:21.898797 20050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:21.908643 20050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:21.909961 20050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:21.911267 20050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:21.923890 20056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:21.923913 20058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:21.924508 20055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:21.924760 20050 server_base.cc:1034] running on GCE node
I20260620 07:02:21.925393 20050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:21.926309 20050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:21.927528 20050 hybrid_clock.cc:648] HybridClock initialized: now 1781938941927474 us; error 40 us; skew 500 ppm
I20260620 07:02:21.927841 20050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:21 dist-test-slave-mz20 krb5kdc[19677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:21.930894 20050 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 07:02:21.932272 20050 webserver.cc:460] Webserver started at http://127.0.97.3:33029/ using document root <none> and password file <none>
I20260620 07:02:21.932883 20050 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:21.932976 20050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:21.933215 20050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:21.935981 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "b10c00f4fd6e42a0b988a8af7a7219de"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "5448bff5706cba5a923e5077ebe5a58a"
server_key_iv: "5dff43f8fd7a9f2942a462d8f6f61c9e"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.936685 20050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b10c00f4fd6e42a0b988a8af7a7219de"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "5448bff5706cba5a923e5077ebe5a58a"
server_key_iv: "5dff43f8fd7a9f2942a462d8f6f61c9e"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.941479 20050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 07:02:21.944767 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.946022 20050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 07:02:21.946169 20050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "b10c00f4fd6e42a0b988a8af7a7219de"
format_stamp: "Formatted at 2026-06-20 07:02:21 on dist-test-slave-mz20"
server_key: "5448bff5706cba5a923e5077ebe5a58a"
server_key_iv: "5dff43f8fd7a9f2942a462d8f6f61c9e"
server_key_version: "encryptionkey@0"
I20260620 07:02:21.946346 20050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:21.958106 20050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:21.961985 20050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:21.962262 20050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:21.963222 20050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:21.964466 20050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:21.964542 20050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.964655 20050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:21.964705 20050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:21.994437 20050 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44841
I20260620 07:02:21.994488 20177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44841 every 8 connection(s)
I20260620 07:02:21.995678 20050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:22.002928 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20050
I20260620 07:02:22.003104 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:22.003665 388 external_mini_cluster.cc:1428] Setting key 7e6295df5a469070b8147a5dc1cf8fa0
Jun 20 07:02:22 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:22.015950 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.998179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33565 (local address 127.0.97.62:43765)
0620 07:02:21.998399 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:21.998405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:21.999558 (+ 1153us) server_negotiation.cc:407] Connection header received
0620 07:02:22.001555 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.001559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.001634 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.001722 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:22.003437 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.004280 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.006577 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.007066 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.007180 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.010631 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:22.010636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:22.010640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:22.010679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:22.012944 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.013639 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.013645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.013649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.013781 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.014211 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.014214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.014216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.014500 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:22.014682 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.015372 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.015619 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 07:02:22.016865 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:21.998659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43765 (local address 127.0.97.3:33565)
0620 07:02:21.999394 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:21.999435 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.001234 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.001983 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.001992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.002511 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:22.003248 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.003274 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.004469 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.004473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.006370 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.006393 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.007283 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.007290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.007520 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.008188 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:02:22.008213 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:22.010380 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:22.013079 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:22.013089 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:22.013106 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:22.013524 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:22.013885 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:02:22.013918 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:02:22.013921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:02:22.014102 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:02:22.014653 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:22.014662 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:22.015018 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.015264 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.015802 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":313,"threads_started":1}
I20260620 07:02:22.018339 20178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43765
I20260620 07:02:22.018671 20178 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:22.019436 20178 heartbeater.cc:511] Master 127.0.97.62:43765 requested a full tablet report, sending...
I20260620 07:02:22.020699 19721 ts_manager.cc:194] Registered new tserver with Master: b10c00f4fd6e42a0b988a8af7a7219de (127.0.97.3:44841)
I20260620 07:02:22.021513 19721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33565
I20260620 07:02:22.031433 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:22.034559 19911 heartbeater.cc:503] Master 127.0.97.62:43765 was elected leader, sending a full tablet report...
I20260620 07:02:22.047206 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.035500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.97.62:43765)
0620 07:02:22.035686 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.035692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.036025 (+ 333us) server_negotiation.cc:407] Connection header received
0620 07:02:22.036194 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.036199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.036273 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.036375 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:22.037679 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.038450 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.039777 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.040214 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.040347 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.042494 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:22.042500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:22.042503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:22.042582 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:22.044901 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.045525 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.045529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.045532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.045644 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.045953 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.045957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.045959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.046302 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:22.046496 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.046806 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.046976 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:02:22.051152 19721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:22.058319 19721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:22.084607 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.071797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.0.97.1:36349)
0620 07:02:22.072484 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.072495 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.076271 (+ 3776us) server_negotiation.cc:407] Connection header received
0620 07:02:22.076532 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.076540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.076779 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.076974 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:22.077947 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079472 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.081585 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.082782 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.082909 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.083154 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.084120 (+ 966us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.084293 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":313,"threads_started":1}
I20260620 07:02:22.084610 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.071518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36349 (local address 127.0.0.1:47162)
0620 07:02:22.075582 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:22.076197 (+ 615us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.076358 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.077264 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.077269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.077297 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:22.077734 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.077745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079615 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.079623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.081422 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.081444 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.083305 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.083312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.083918 (+ 606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.083966 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.084056 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.084184 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3888,"mutex_wait_us":2,"thread_start_us":4191,"threads_started":1}
I20260620 07:02:22.084731 20193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.074913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37247 (local address 127.0.0.1:36738)
0620 07:02:22.075965 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:22.075982 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.076090 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.077510 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.077514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.077537 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:22.077966 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.077977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079448 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.079455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.082030 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.082045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.084001 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.084007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.084158 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.084189 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.084257 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.084374 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":137,"thread_start_us":794,"threads_started":1}
I20260620 07:02:22.084765 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.071790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.97.3:44841)
0620 07:02:22.075017 (+ 3227us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.075022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.076121 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 07:02:22.076235 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.076243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.076492 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.076672 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:22.077628 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079243 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.082204 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.083311 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.083432 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.083635 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.084292 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.084456 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3091,"thread_start_us":2935,"threads_started":1}
I20260620 07:02:22.085027 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.071272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44841 (local address 127.0.0.1:41282)
0620 07:02:22.075582 (+ 4310us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:22.075656 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.075874 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.076978 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.076985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.077013 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:22.077412 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.077433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079356 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.079360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.082034 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.082045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.083499 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.083505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.083703 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.083756 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.083885 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.084047 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4025,"mutex_wait_us":130,"thread_start_us":4753,"threads_started":1}
I20260620 07:02:22.085597 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.075546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.0.97.2:37247)
0620 07:02:22.076919 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.076926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.076949 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:02:22.077031 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.077038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.077264 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.077510 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:22.078096 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.079132 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.082186 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.083573 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.083691 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.083860 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.085172 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.085332 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":1651,"threads_started":1}
I20260620 07:02:22.088779 19980 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89817018634e1aa23c40a75efd13c5 (DEFAULT_TABLE table=test-table [id=7804699e63ba4f768ce432a48bba7ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:22.088862 20113 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89817018634e1aa23c40a75efd13c5 (DEFAULT_TABLE table=test-table [id=7804699e63ba4f768ce432a48bba7ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:22.088862 19846 tablet_service.cc:1467] Processing CreateTablet for tablet 0e89817018634e1aa23c40a75efd13c5 (DEFAULT_TABLE table=test-table [id=7804699e63ba4f768ce432a48bba7ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:22.089991 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89817018634e1aa23c40a75efd13c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:22.090222 19846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89817018634e1aa23c40a75efd13c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:22.090222 20113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e89817018634e1aa23c40a75efd13c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:22.101218 20196 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Bootstrap starting.
I20260620 07:02:22.104157 20197 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Bootstrap starting.
I20260620 07:02:22.104952 20198 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Bootstrap starting.
I20260620 07:02:22.105026 20196 tablet_bootstrap.cc:654] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:22.106505 20196 log.cc:824] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:22.108245 20198 tablet_bootstrap.cc:654] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:22.109238 20197 tablet_bootstrap.cc:654] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:22.109421 20198 log.cc:824] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:22.109686 20196 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: No bootstrap required, opened a new log
I20260620 07:02:22.109977 20196 ts_tablet_manager.cc:1397] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 07:02:22.110266 20197 log.cc:824] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:22.112401 20198 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: No bootstrap required, opened a new log
I20260620 07:02:22.112650 20198 ts_tablet_manager.cc:1397] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 07:02:22.114688 20197 tablet_bootstrap.cc:492] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: No bootstrap required, opened a new log
I20260620 07:02:22.114987 20197 ts_tablet_manager.cc:1397] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260620 07:02:22.120153 20196 raft_consensus.cc:348] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.120621 20196 raft_consensus.cc:374] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:22.120774 20196 raft_consensus.cc:729] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd01b07edd1479696b6d92e53d079f8, State: Initialized, Role: FOLLOWER
I20260620 07:02:22.121459 20196 consensus_queue.cc:260] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.122648 20045 heartbeater.cc:503] Master 127.0.97.62:43765 was elected leader, sending a full tablet report...
I20260620 07:02:22.122598 20197 raft_consensus.cc:348] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.123003 20197 raft_consensus.cc:374] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:22.123100 20197 raft_consensus.cc:729] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a6756646aee4665a732a4f92687d3df, State: Initialized, Role: FOLLOWER
I20260620 07:02:22.123324 20196 ts_tablet_manager.cc:1428] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 07:02:22.123656 20197 consensus_queue.cc:260] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.124222 20198 raft_consensus.cc:348] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.124703 20198 raft_consensus.cc:374] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:22.124930 20198 raft_consensus.cc:729] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10c00f4fd6e42a0b988a8af7a7219de, State: Initialized, Role: FOLLOWER
I20260620 07:02:22.125420 20197 ts_tablet_manager.cc:1428] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 07:02:22.125674 20198 consensus_queue.cc:260] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.126762 20178 heartbeater.cc:503] Master 127.0.97.62:43765 was elected leader, sending a full tablet report...
I20260620 07:02:22.128643 20198 ts_tablet_manager.cc:1428] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260620 07:02:22.228194 20203 raft_consensus.cc:482] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:22.228482 20203 raft_consensus.cc:504] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.230331 20203 leader_election.cc:288] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b10c00f4fd6e42a0b988a8af7a7219de (127.0.97.3:44841), bfd01b07edd1479696b6d92e53d079f8 (127.0.97.2:37247)
I20260620 07:02:22.236720 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.230795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36643 (local address 127.0.97.3:44841)
0620 07:02:22.231040 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.231047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.231461 (+ 414us) server_negotiation.cc:407] Connection header received
0620 07:02:22.231649 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.231656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.231719 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.231789 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:22.232319 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.233236 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.235081 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.235880 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.235994 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.236093 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.236359 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.236489 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 07:02:22.236806 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.230684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44841 (local address 127.0.97.1:36643)
0620 07:02:22.231369 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:22.231383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.231503 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.231832 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.231837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.231865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:22.232194 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.232203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.233359 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.233363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.234915 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.234929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.236079 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.236086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.236218 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.236241 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.236320 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.236449 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":947,"threads_started":1}
I20260620 07:02:22.237437 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.230724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37247 (local address 127.0.97.1:41303)
0620 07:02:22.231935 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:22.231949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:22.232078 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:22.232373 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:22.232377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:22.232399 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:22.232671 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.232679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.233731 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.233735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.235354 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:22.235368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.236698 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.236705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:22.236844 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.236941 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 07:02:22.237082 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 07:02:22.237197 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":349,"threads_started":1}
I20260620 07:02:22.237505 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.230803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41303 (local address 127.0.97.2:37247)
0620 07:02:22.231074 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.231079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.232036 (+ 957us) server_negotiation.cc:407] Connection header received
0620 07:02:22.232208 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.232214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.232276 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.232347 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:22.232841 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.233597 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.235515 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.236426 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.236557 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.236647 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.237116 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.237255 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260620 07:02:22.237980 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e89817018634e1aa23c40a75efd13c5" candidate_uuid: "5a6756646aee4665a732a4f92687d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" is_pre_election: true
I20260620 07:02:22.238399 20000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e89817018634e1aa23c40a75efd13c5" candidate_uuid: "5a6756646aee4665a732a4f92687d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd01b07edd1479696b6d92e53d079f8" is_pre_election: true
I20260620 07:02:22.238502 20133 raft_consensus.cc:2454] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a6756646aee4665a732a4f92687d3df in term 0.
I20260620 07:02:22.238847 20000 raft_consensus.cc:2454] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a6756646aee4665a732a4f92687d3df in term 0.
I20260620 07:02:22.239238 19801 leader_election.cc:302] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a6756646aee4665a732a4f92687d3df, b10c00f4fd6e42a0b988a8af7a7219de; no voters:
I20260620 07:02:22.239665 20203 raft_consensus.cc:2781] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:22.239782 20203 raft_consensus.cc:482] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:22.239852 20203 raft_consensus.cc:3037] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:22.242007 20203 raft_consensus.cc:504] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.242635 20203 leader_election.cc:288] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [CANDIDATE]: Term 1 election: Requested vote from peers b10c00f4fd6e42a0b988a8af7a7219de (127.0.97.3:44841), bfd01b07edd1479696b6d92e53d079f8 (127.0.97.2:37247)
I20260620 07:02:22.243129 20133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e89817018634e1aa23c40a75efd13c5" candidate_uuid: "5a6756646aee4665a732a4f92687d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10c00f4fd6e42a0b988a8af7a7219de"
I20260620 07:02:22.243307 20000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e89817018634e1aa23c40a75efd13c5" candidate_uuid: "5a6756646aee4665a732a4f92687d3df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfd01b07edd1479696b6d92e53d079f8"
I20260620 07:02:22.243346 20133 raft_consensus.cc:3037] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:22.243513 20000 raft_consensus.cc:3037] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:22.246578 20133 raft_consensus.cc:2454] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a6756646aee4665a732a4f92687d3df in term 1.
I20260620 07:02:22.246580 20000 raft_consensus.cc:2454] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a6756646aee4665a732a4f92687d3df in term 1.
I20260620 07:02:22.247064 19800 leader_election.cc:302] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a6756646aee4665a732a4f92687d3df, bfd01b07edd1479696b6d92e53d079f8; no voters:
W20260620 07:02:22.247192 20179 tablet.cc:2369] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:22.247432 20203 raft_consensus.cc:2781] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:22.247910 20203 raft_consensus.cc:686] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 1 LEADER]: Becoming Leader. State: Replica: 5a6756646aee4665a732a4f92687d3df, State: Running, Role: LEADER
I20260620 07:02:22.248373 20203 consensus_queue.cc:237] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } }
I20260620 07:02:22.252815 19722 catalog_manager.cc:5495] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a6756646aee4665a732a4f92687d3df (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5a6756646aee4665a732a4f92687d3df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a6756646aee4665a732a4f92687d3df" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 } health_report { overall_health: UNKNOWN } } }
W20260620 07:02:22.256989 19912 tablet.cc:2369] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 07:02:22.257514 20046 tablet.cc:2369] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:22.278497 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.272989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.0.97.1:36349)
0620 07:02:22.273195 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.273202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.273268 (+ 66us) server_negotiation.cc:407] Connection header received
0620 07:02:22.273497 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.273501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.273602 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.273785 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:22.274163 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.274923 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.276360 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.276817 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.276923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.277420 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:22.277929 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:22.278059 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.278160 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.278246 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:02:22.286789 20133 raft_consensus.cc:1261] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 1 FOLLOWER]: Refusing update from remote peer 5a6756646aee4665a732a4f92687d3df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:22.286789 20000 raft_consensus.cc:1261] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 1 FOLLOWER]: Refusing update from remote peer 5a6756646aee4665a732a4f92687d3df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:22.287910 20207 consensus_queue.cc:1035] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10c00f4fd6e42a0b988a8af7a7219de" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:22.288241 20203 consensus_queue.cc:1035] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfd01b07edd1479696b6d92e53d079f8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:22.297660 20210 mvcc.cc:205] Tried to move back new op lower bound from 7298821907595874304 to 7298821907453644800. Current Snapshot: MvccSnapshot[applied={T|T < 7298821907595874304}]
I20260620 07:02:22.313450 19722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:22.313745 19722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:22.318147 19722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d258a4d89984d5daf0118217b8c14f4: Sending DeleteTablet for 3 replicas of tablet 0e89817018634e1aa23c40a75efd13c5
I20260620 07:02:22.319271 20113 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89817018634e1aa23c40a75efd13c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41282
I20260620 07:02:22.319411 19846 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89817018634e1aa23c40a75efd13c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47162
I20260620 07:02:22.319947 19980 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e89817018634e1aa23c40a75efd13c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36738
I20260620 07:02:22.322183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19783
I20260620 07:02:22.331103 20222 tablet_replica.cc:330] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: stopping tablet replica
I20260620 07:02:22.332810 20222 raft_consensus.cc:2229] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:22.334302 20222 raft_consensus.cc:2258] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:22.335191 20220 tablet_replica.cc:330] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: stopping tablet replica
I20260620 07:02:22.336180 20220 raft_consensus.cc:2229] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:22.338585 20222 ts_tablet_manager.cc:1905] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:22.343264 20220 raft_consensus.cc:2258] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:22.346045 20222 ts_tablet_manager.cc:1918] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:22.346237 20222 log.cc:1196] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/0e89817018634e1aa23c40a75efd13c5
I20260620 07:02:22.346810 20222 ts_tablet_manager.cc:1939] T 0e89817018634e1aa23c40a75efd13c5 P bfd01b07edd1479696b6d92e53d079f8: Deleting consensus metadata
I20260620 07:02:22.347465 20221 tablet_replica.cc:330] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: stopping tablet replica
I20260620 07:02:22.347879 20221 raft_consensus.cc:2229] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:22.348443 20221 raft_consensus.cc:2258] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:22.348821 19707 catalog_manager.cc:4841] TS bfd01b07edd1479696b6d92e53d079f8 (127.0.97.2:37247): tablet 0e89817018634e1aa23c40a75efd13c5 (table test-table [id=7804699e63ba4f768ce432a48bba7ae0]) successfully deleted
I20260620 07:02:22.349910 20220 ts_tablet_manager.cc:1905] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 07:02:22 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 07:02:22.352617 20221 ts_tablet_manager.cc:1905] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:22.355072 20220 ts_tablet_manager.cc:1918] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:22.355226 20220 log.cc:1196] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/0e89817018634e1aa23c40a75efd13c5
I20260620 07:02:22.355710 20220 ts_tablet_manager.cc:1939] T 0e89817018634e1aa23c40a75efd13c5 P b10c00f4fd6e42a0b988a8af7a7219de: Deleting consensus metadata
I20260620 07:02:22.357560 19708 catalog_manager.cc:4841] TS b10c00f4fd6e42a0b988a8af7a7219de (127.0.97.3:44841): tablet 0e89817018634e1aa23c40a75efd13c5 (table test-table [id=7804699e63ba4f768ce432a48bba7ae0]) successfully deleted
I20260620 07:02:22.360169 20221 ts_tablet_manager.cc:1918] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:22.360383 20221 log.cc:1196] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781938634772379-388-0/minicluster-data/ts-0/wal/wals/0e89817018634e1aa23c40a75efd13c5
I20260620 07:02:22.360909 20221 ts_tablet_manager.cc:1939] T 0e89817018634e1aa23c40a75efd13c5 P 5a6756646aee4665a732a4f92687d3df: Deleting consensus metadata
I20260620 07:02:22.362772 19706 catalog_manager.cc:4841] TS 5a6756646aee4665a732a4f92687d3df (127.0.97.1:36349): tablet 0e89817018634e1aa23c40a75efd13c5 (table test-table [id=7804699e63ba4f768ce432a48bba7ae0]) successfully deleted
I20260620 07:02:22.363168 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.323751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.0.97.1:36349)
0620 07:02:22.323950 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.323957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.328081 (+ 4124us) server_negotiation.cc:407] Connection header received
0620 07:02:22.328359 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.328365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.328473 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.328582 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:22.332739 (+ 4157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.334125 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.336535 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.337361 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.337481 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.356696 (+ 19215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:22.356727 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:22.356745 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:22.356806 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:22.360108 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.360965 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.360975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.360978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.361175 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.361563 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.361567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.361571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.361933 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:22.362223 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.362529 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.362796 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:02:22.363814 19906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:22.473022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19783
I20260620 07:02:22.487671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19917
Jun 20 07:02:22 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 07:02:22.500918 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.488305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.0.97.2:37247)
0620 07:02:22.488453 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.488457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.488474 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:02:22.488593 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.488596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.488670 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.488750 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:22.489932 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.490703 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.491842 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.492210 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.492301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.496119 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:22.496134 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:22.496150 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:22.496185 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:22.498768 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.499374 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.499379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.499382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.499495 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.499758 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.499761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.499763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.500102 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:22.500314 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.500510 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.500665 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 07:02:22.501540 20040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:22.589444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19917
I20260620 07:02:22.603374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20050
Jun 20 07:02:22 dist-test-slave-mz20 krb5kdc[19677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 07:02:22.617509 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:22.604113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.97.3:44841)
0620 07:02:22.604302 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:22.604308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:22.604328 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:02:22.604456 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:22.604460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:22.604526 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:22.604611 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:22.605916 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.606688 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.607886 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:22.608281 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:22.608384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:22.612464 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:22.612479 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:22.612495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:22.612532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:22.614960 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.615581 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.615590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.615592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.615730 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:22.616061 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:22.616065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:22.616067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:22.616434 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:22.616702 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:22.616995 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 07:02:22.617225 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 07:02:22.618132 20173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:22.705494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20050
I20260620 07:02:22.718712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 19690
I20260620 07:02:22.719697 19752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:22.830809 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 19690
2026-06-20T07:02:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4100 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T07:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:22Z Disabled control of system clock
I20260620 07:02:22.900163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38215
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:23.220129 20231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:23.220419 20231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:23.220521 20231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:23.227856 20231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:23.227974 20231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:23.228012 20231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:23.228088 20231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:23.228148 20231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:23.237375 20231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:23.238667 20231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:23.240458 20231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:23.252290 20236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:23.252372 20239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:23.252341 20237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:23.253154 20231 server_base.cc:1034] running on GCE node
I20260620 07:02:23.253895 20231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:23.255252 20231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:23.256458 20231 hybrid_clock.cc:648] HybridClock initialized: now 1781938943256402 us; error 44 us; skew 500 ppm
I20260620 07:02:23.256839 20231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:23.258912 20231 webserver.cc:460] Webserver started at http://127.0.97.62:46233/ using document root <none> and password file <none>
I20260620 07:02:23.259521 20231 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:23.259598 20231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:23.259840 20231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:23.262658 20231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "9b0e5977c5074836a39285f81aca7efa"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "ff34f9c4b8c58f022a934836a3662aa0"
server_key_iv: "9ca0e30b27087a680b66256da5078570"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.263374 20231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b0e5977c5074836a39285f81aca7efa"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "ff34f9c4b8c58f022a934836a3662aa0"
server_key_iv: "9ca0e30b27087a680b66256da5078570"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.268144 20231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 07:02:23.271467 20245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:23.272698 20231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:23.272857 20231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "9b0e5977c5074836a39285f81aca7efa"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "ff34f9c4b8c58f022a934836a3662aa0"
server_key_iv: "9ca0e30b27087a680b66256da5078570"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.273043 20231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:23.299692 20231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:23.300552 20231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:23.300814 20231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:23.317586 20231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38215
I20260620 07:02:23.317610 20296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38215 every 8 connection(s)
I20260620 07:02:23.318910 20231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:23.322659 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:23.322748 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20231
I20260620 07:02:23.322914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:23.323365 388 external_mini_cluster.cc:1428] Setting key d51ed3ee92efa52800b9621c894c008a
I20260620 07:02:23.337033 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: Bootstrap starting.
I20260620 07:02:23.338739 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:23.325571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.0.97.62:38215)
0620 07:02:23.326701 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:23.326727 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:23.326777 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:02:23.329322 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:23.329352 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:23.329897 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:23.330269 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:23.330899 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.332177 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.334104 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.334937 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.335073 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:23.336346 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:23.336354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:23.336369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:23.336921 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:23.336999 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:23.337029 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:23.337121 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:23.337357 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 07:02:23.337679 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":396,"threads_started":1}
I20260620 07:02:23.340405 20297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:23.341490 20297 log.cc:824] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:23.344492 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: No bootstrap required, opened a new log
I20260620 07:02:23.351928 20297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } }
I20260620 07:02:23.352409 20297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:23.352527 20297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0e5977c5074836a39285f81aca7efa, State: Initialized, Role: FOLLOWER
I20260620 07:02:23.353169 20297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } }
I20260620 07:02:23.353415 20297 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:23.353515 20297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:23.353641 20297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:23.355847 20297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } }
I20260620 07:02:23.356312 20297 leader_election.cc:302] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b0e5977c5074836a39285f81aca7efa; no voters:
I20260620 07:02:23.356858 20297 leader_election.cc:288] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:23.357029 20302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:23.357677 20302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [term 1 LEADER]: Becoming Leader. State: Replica: 9b0e5977c5074836a39285f81aca7efa, State: Running, Role: LEADER
I20260620 07:02:23.358193 20302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } }
I20260620 07:02:23.358871 20297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:23.360963 20304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b0e5977c5074836a39285f81aca7efa. Latest consensus state: current_term: 1 leader_uuid: "9b0e5977c5074836a39285f81aca7efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } } }
I20260620 07:02:23.361395 20304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:23.361398 20303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b0e5977c5074836a39285f81aca7efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b0e5977c5074836a39285f81aca7efa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38215 } } }
I20260620 07:02:23.362185 20303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:23.363145 20311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:23.368158 20311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:23.376798 20311 catalog_manager.cc:1347] Generated new cluster ID: fc54c34796e9447a9d717cc2da67199e
I20260620 07:02:23.376907 20311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:23.390771 20311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:23.392485 20311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:23.403959 20311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: Generated new TSK 0
I20260620 07:02:23.405001 20311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:23.417212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--builtin_ntp_servers=127.0.97.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 07:02:23.742465 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:23.742878 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:23.743034 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:23.751327 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:23.751422 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:23.751529 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 07:02:23.760648 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:23.761790 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:23.763710 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:23.775899 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:23.775987 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:23.775909 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:23.777421 20321 server_base.cc:1034] running on GCE node
I20260620 07:02:23.778030 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:23.779351 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:23.780566 20321 hybrid_clock.cc:648] HybridClock initialized: now 1781938943780518 us; error 41 us; skew 500 ppm
I20260620 07:02:23.780879 20321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:23.783313 20321 webserver.cc:460] Webserver started at http://127.0.97.1:43055/ using document root <none> and password file <none>
I20260620 07:02:23.783999 20321 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:23.784106 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:23.784410 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:23.787353 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data/instance:
uuid: "cebae3ecd2cd4278ab8d1aaaa9774548"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "206ebef6a6f69c09b39b5385979e5f2a"
server_key_iv: "9947567a1bc5557ab4341292c6cb64dd"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.788122 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cebae3ecd2cd4278ab8d1aaaa9774548"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "206ebef6a6f69c09b39b5385979e5f2a"
server_key_iv: "9947567a1bc5557ab4341292c6cb64dd"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.793226 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:23.796691 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:23.797997 20321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:23.798168 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal
uuid: "cebae3ecd2cd4278ab8d1aaaa9774548"
format_stamp: "Formatted at 2026-06-20 07:02:23 on dist-test-slave-mz20"
server_key: "206ebef6a6f69c09b39b5385979e5f2a"
server_key_iv: "9947567a1bc5557ab4341292c6cb64dd"
server_key_version: "encryptionkey@0"
I20260620 07:02:23.798374 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:23.819999 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:23.820811 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:23.821064 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:23.822008 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:23.823575 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:23.823670 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:23.823776 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:23.823841 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:23.854060 20321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39647
I20260620 07:02:23.854127 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39647 every 8 connection(s)
I20260620 07:02:23.855314 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/data/info.pb
I20260620 07:02:23.863837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20321
I20260620 07:02:23.864006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-0/wal/instance
I20260620 07:02:23.864547 388 external_mini_cluster.cc:1428] Setting key 0a4494dc8cdcb62399b179afbdb47500
I20260620 07:02:23.867328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--builtin_ntp_servers=127.0.97.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:23.867931 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:23.857675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40753 (local address 127.0.97.62:38215)
0620 07:02:23.858099 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:23.858104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:23.859339 (+ 1235us) server_negotiation.cc:407] Connection header received
0620 07:02:23.861116 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:23.861120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:23.861188 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:23.861273 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:23.862351 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.863216 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.865330 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.865766 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.865871 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:23.867232 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:23.867237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:23.867241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:23.867258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:23.867303 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:23.867316 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:23.867395 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:23.867622 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:02:23.867696 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":212,"threads_started":1}
I20260620 07:02:23.869227 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:23.858171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38215 (local address 127.0.97.1:40753)
0620 07:02:23.859162 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:23.859208 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:23.860860 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:23.861515 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:23.861523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:23.861607 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:23.862190 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:23.862208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.863364 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.863367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:23.865196 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:23.865209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:23.865983 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:23.865991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:23.866216 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:23.866885 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:23.866889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:23.866949 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:23.867052 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:23.867413 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:23.867423 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:23.867432 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:02:23.867607 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 07:02:23.867976 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":339,"threads_started":1}
I20260620 07:02:23.870671 20448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38215
I20260620 07:02:23.870995 20448 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:23.871706 20448 heartbeater.cc:511] Master 127.0.97.62:38215 requested a full tablet report, sending...
I20260620 07:02:23.873670 20262 ts_manager.cc:194] Registered new tserver with Master: cebae3ecd2cd4278ab8d1aaaa9774548 (127.0.97.1:39647)
I20260620 07:02:23.875545 20262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40753
W20260620 07:02:24.184587 20452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:24.184861 20452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:24.184989 20452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:24.192246 20452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:24.192353 20452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:24.192493 20452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 07:02:24.201620 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:24.202998 20452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:24.204689 20452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:24.216856 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:24.217125 20460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:24.217147 20458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:24.218003 20452 server_base.cc:1034] running on GCE node
I20260620 07:02:24.218643 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:24.219592 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:24.220793 20452 hybrid_clock.cc:648] HybridClock initialized: now 1781938944220735 us; error 63 us; skew 500 ppm
I20260620 07:02:24.221096 20452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:24.223511 20452 webserver.cc:460] Webserver started at http://127.0.97.2:37289/ using document root <none> and password file <none>
I20260620 07:02:24.224241 20452 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:24.224350 20452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:24.224663 20452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:24.227617 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data/instance:
uuid: "3c0328c3256c4ec691e7167f811ec870"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "7690480631975f71de09c956c44c7bdb"
server_key_iv: "a0d4ee6174d21490309a4d908eaf4031"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.228389 20452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3c0328c3256c4ec691e7167f811ec870"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "7690480631975f71de09c956c44c7bdb"
server_key_iv: "a0d4ee6174d21490309a4d908eaf4031"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.233536 20452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:24.236889 20466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.238094 20452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:24.238240 20452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal
uuid: "3c0328c3256c4ec691e7167f811ec870"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "7690480631975f71de09c956c44c7bdb"
server_key_iv: "a0d4ee6174d21490309a4d908eaf4031"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.238446 20452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:24.253150 20452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:24.253983 20452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:24.254235 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:24.255236 20452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:24.256698 20452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:24.256793 20452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.256903 20452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:24.256970 20452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.286923 20452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40753
I20260620 07:02:24.286984 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40753 every 8 connection(s)
I20260620 07:02:24.288182 20452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/data/info.pb
I20260620 07:02:24.291378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20452
I20260620 07:02:24.291527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal/instance
I20260620 07:02:24.292061 388 external_mini_cluster.cc:1428] Setting key 5cba622c1bbd755bf423e37cee6651f1
I20260620 07:02:24.295039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--builtin_ntp_servers=127.0.97.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 07:02:24.302076 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.290653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52067 (local address 127.0.97.62:38215)
0620 07:02:24.290858 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.290865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.292421 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 07:02:24.294530 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.294559 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.294633 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.294708 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:24.296069 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.296835 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.299081 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.299517 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.299622 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.301243 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:24.301248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:24.301251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:24.301271 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:24.301305 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:24.301318 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:24.301396 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.301648 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.301806 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 07:02:24.303437 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.291208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38215 (local address 127.0.97.2:52067)
0620 07:02:24.292249 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.292298 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.294230 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.294977 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.294989 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.295109 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:24.295853 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.295876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.296992 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.296997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.298886 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.298903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.299758 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.299765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.300054 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.300856 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:24.300859 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:24.300916 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:24.301021 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:24.301406 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:24.301415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:24.301423 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.301608 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.302081 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":317,"threads_started":1}
I20260620 07:02:24.304955 20579 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38215
I20260620 07:02:24.305266 20579 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:24.306011 20579 heartbeater.cc:511] Master 127.0.97.62:38215 requested a full tablet report, sending...
I20260620 07:02:24.307482 20262 ts_manager.cc:194] Registered new tserver with Master: 3c0328c3256c4ec691e7167f811ec870 (127.0.97.2:40753)
I20260620 07:02:24.308290 20262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52067
W20260620 07:02:24.621403 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:24.621742 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:24.621861 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:24.629411 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:24.629575 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:24.629707 20582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 07:02:24.638967 20582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:24.640379 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:24.642390 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:24.656111 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:24.656082 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:24.656046 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:24.656808 20582 server_base.cc:1034] running on GCE node
I20260620 07:02:24.657398 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:24.658376 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:24.659579 20582 hybrid_clock.cc:648] HybridClock initialized: now 1781938944659522 us; error 73 us; skew 500 ppm
I20260620 07:02:24.659909 20582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 07:02:24.662220 20582 webserver.cc:460] Webserver started at http://127.0.97.3:43489/ using document root <none> and password file <none>
I20260620 07:02:24.662955 20582 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:24.663056 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:24.663370 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:24.666344 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data/instance:
uuid: "68def7d0cc2240718f3a49de1980c38b"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "79f75c7b6d252a84bf5a2eda0d042218"
server_key_iv: "ac8520c95d9e43b36f97547377807b47"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.667191 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal/instance:
uuid: "68def7d0cc2240718f3a49de1980c38b"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "79f75c7b6d252a84bf5a2eda0d042218"
server_key_iv: "ac8520c95d9e43b36f97547377807b47"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.672501 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:24.676107 20596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.677628 20582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:02:24.677816 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal
uuid: "68def7d0cc2240718f3a49de1980c38b"
format_stamp: "Formatted at 2026-06-20 07:02:24 on dist-test-slave-mz20"
server_key: "79f75c7b6d252a84bf5a2eda0d042218"
server_key_iv: "ac8520c95d9e43b36f97547377807b47"
server_key_version: "encryptionkey@0"
I20260620 07:02:24.678030 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:24.693516 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:24.694346 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:24.694623 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:24.695592 20582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 07:02:24.697086 20582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 07:02:24.697175 20582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.697257 20582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 07:02:24.697309 20582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:24.728637 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39329
I20260620 07:02:24.728736 20708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39329 every 8 connection(s)
I20260620 07:02:24.730160 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/data/info.pb
I20260620 07:02:24.731545 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20582
I20260620 07:02:24.731712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal/instance
I20260620 07:02:24.732175 388 external_mini_cluster.cc:1428] Setting key 53dd7651470f00ae957004f0272e0832
I20260620 07:02:24.742825 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.732808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46795 (local address 127.0.97.62:38215)
0620 07:02:24.733029 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.733035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.734632 (+ 1597us) server_negotiation.cc:407] Connection header received
0620 07:02:24.736513 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.736519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.736604 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.736689 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:24.737747 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.738462 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.739970 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.740390 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.740499 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.741948 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:24.741954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:24.741959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:24.741980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:24.742048 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:24.742063 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:24.742138 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.742342 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.742443 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:02:24.743822 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.733349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38215 (local address 127.0.97.3:46795)
0620 07:02:24.734477 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.734513 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.736247 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.736936 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.736945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.737034 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:02:24.737570 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.737586 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.738651 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.738656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.739839 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.739850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.740598 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.740605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.740905 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.741625 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 07:02:24.741628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:02:24.741669 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 07:02:24.741769 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:02:24.742150 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:02:24.742161 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:02:24.742170 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.742321 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.742736 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":489,"threads_started":1}
I20260620 07:02:24.745291 20709 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38215
I20260620 07:02:24.745573 20709 heartbeater.cc:464] Registering TS with master...
I20260620 07:02:24.746451 20709 heartbeater.cc:511] Master 127.0.97.62:38215 requested a full tablet report, sending...
I20260620 07:02:24.747956 20262 ts_manager.cc:194] Registered new tserver with Master: 68def7d0cc2240718f3a49de1980c38b (127.0.97.3:39329)
I20260620 07:02:24.748719 20262 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:46795
I20260620 07:02:24.759718 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 07:02:24.769583 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.763808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.97.62:38215)
0620 07:02:24.764051 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.764058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.764389 (+ 331us) server_negotiation.cc:407] Connection header received
0620 07:02:24.764576 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.764580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.764661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.764749 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:24.765259 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.765958 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.767430 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.767855 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.767952 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.768878 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:24.768883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:24.768886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:24.768904 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:24.768934 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:24.768947 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:24.769012 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.769204 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.769279 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 07:02:24.773312 20262 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 07:02:24.780238 20262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 07:02:24.818199 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.792309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40753 (local address 127.0.0.1:56326)
0620 07:02:24.801198 (+ 8889us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.801249 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.801503 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.811254 (+ 9751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.811259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.811283 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:24.811738 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.811751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813215 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.813220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.815881 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.815894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817201 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.817381 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.817426 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.817559 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.817788 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8564,"mutex_wait_us":137,"thread_start_us":17163,"threads_started":1}
I20260620 07:02:24.818286 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.792386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39329 (local address 127.0.0.1:54258)
0620 07:02:24.809747 (+ 17361us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.809772 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.809927 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.810794 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.810802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.810832 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:24.811237 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.811250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813399 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.813404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.815879 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.815894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817557 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.817738 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.817768 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.817853 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.817979 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17198,"mutex_wait_us":70,"thread_start_us":16971,"threads_started":1}
I20260620 07:02:24.818578 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.792443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.0.97.2:40753)
0620 07:02:24.810748 (+ 18305us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.810757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.810780 (+ 23us) server_negotiation.cc:407] Connection header received
0620 07:02:24.810859 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.810865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.811050 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.811254 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:24.811917 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813089 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.816109 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817019 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817163 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.817414 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.817888 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.818082 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17356,"thread_start_us":17588,"threads_started":1}
I20260620 07:02:24.819231 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.793030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54258 (local address 127.0.97.3:39329)
0620 07:02:24.809290 (+ 16260us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.809300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.810063 (+ 763us) server_negotiation.cc:407] Connection header received
0620 07:02:24.810180 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.810189 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.810483 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.811460 (+ 977us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:24.811738 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813269 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.816116 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817396 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817510 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.817774 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.818650 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.818883 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16051,"thread_start_us":15885,"threads_started":1}
I20260620 07:02:24.819278 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.795707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.0.97.1:39647)
0620 07:02:24.810700 (+ 14993us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.810709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.810736 (+ 27us) server_negotiation.cc:407] Connection header received
0620 07:02:24.810831 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.810838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.811050 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.811254 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:24.812162 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813605 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.815942 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817333 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817479 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.817717 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.818818 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.818989 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14801,"thread_start_us":14670,"threads_started":1}
I20260620 07:02:24.818703 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.795029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34702)
0620 07:02:24.809784 (+ 14755us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.809804 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.809954 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.811484 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.811487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.811510 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:24.811962 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.811976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.813741 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.813746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.815698 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.815712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.817851 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.817858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.818036 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.818071 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.818156 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.818293 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14623,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":5708,"threads_started":1}
I20260620 07:02:24.822628 20514 tablet_service.cc:1467] Processing CreateTablet for tablet cc357379ad2142ceac6d3af20678b4ef (DEFAULT_TABLE table=test-table [id=2431d9e7362d481e8c15bf2e813b2a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:24.822643 20644 tablet_service.cc:1467] Processing CreateTablet for tablet cc357379ad2142ceac6d3af20678b4ef (DEFAULT_TABLE table=test-table [id=2431d9e7362d481e8c15bf2e813b2a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:24.822638 20383 tablet_service.cc:1467] Processing CreateTablet for tablet cc357379ad2142ceac6d3af20678b4ef (DEFAULT_TABLE table=test-table [id=2431d9e7362d481e8c15bf2e813b2a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 07:02:24.823848 20514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc357379ad2142ceac6d3af20678b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:24.823942 20644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc357379ad2142ceac6d3af20678b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:24.823942 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc357379ad2142ceac6d3af20678b4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:24.835183 20727 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Bootstrap starting.
I20260620 07:02:24.838027 20728 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: Bootstrap starting.
I20260620 07:02:24.840224 20727 tablet_bootstrap.cc:654] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:24.841495 20728 tablet_bootstrap.cc:654] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:24.841547 20727 log.cc:824] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:24.842738 20728 log.cc:824] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:24.843071 20729 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Bootstrap starting.
I20260620 07:02:24.845057 20727 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: No bootstrap required, opened a new log
I20260620 07:02:24.845314 20727 ts_tablet_manager.cc:1397] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 07:02:24.846359 20728 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: No bootstrap required, opened a new log
I20260620 07:02:24.846498 20729 tablet_bootstrap.cc:654] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:24.846709 20728 ts_tablet_manager.cc:1397] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 07:02:24.847723 20729 log.cc:824] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:24.852927 20729 tablet_bootstrap.cc:492] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: No bootstrap required, opened a new log
I20260620 07:02:24.853214 20729 ts_tablet_manager.cc:1397] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 07:02:24.854389 20727 raft_consensus.cc:348] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.854909 20727 raft_consensus.cc:374] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:24.855010 20727 raft_consensus.cc:729] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0328c3256c4ec691e7167f811ec870, State: Initialized, Role: FOLLOWER
I20260620 07:02:24.855607 20727 consensus_queue.cc:260] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.857085 20727 ts_tablet_manager.cc:1428] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 07:02:24.858759 20579 heartbeater.cc:503] Master 127.0.97.62:38215 was elected leader, sending a full tablet report...
I20260620 07:02:24.858718 20728 raft_consensus.cc:348] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.859151 20728 raft_consensus.cc:374] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:24.859241 20728 raft_consensus.cc:729] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebae3ecd2cd4278ab8d1aaaa9774548, State: Initialized, Role: FOLLOWER
I20260620 07:02:24.859838 20728 consensus_queue.cc:260] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.861362 20448 heartbeater.cc:503] Master 127.0.97.62:38215 was elected leader, sending a full tablet report...
I20260620 07:02:24.862087 20728 ts_tablet_manager.cc:1428] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260620 07:02:24.868373 20729 raft_consensus.cc:348] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.868760 20729 raft_consensus.cc:374] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:24.868858 20729 raft_consensus.cc:729] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68def7d0cc2240718f3a49de1980c38b, State: Initialized, Role: FOLLOWER
I20260620 07:02:24.869426 20729 consensus_queue.cc:260] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.870453 20709 heartbeater.cc:503] Master 127.0.97.62:38215 was elected leader, sending a full tablet report...
I20260620 07:02:24.871048 20729 ts_tablet_manager.cc:1428] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Time spent starting tablet: real 0.016s user 0.009s sys 0.002s
I20260620 07:02:24.968492 20734 raft_consensus.cc:482] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 07:02:24.968792 20734 raft_consensus.cc:504] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.970484 20734 leader_election.cc:288] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0328c3256c4ec691e7167f811ec870 (127.0.97.2:40753), 68def7d0cc2240718f3a49de1980c38b (127.0.97.3:39329)
I20260620 07:02:24.977456 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.971013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51437 (local address 127.0.97.2:40753)
0620 07:02:24.971204 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.971210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.971889 (+ 679us) server_negotiation.cc:407] Connection header received
0620 07:02:24.972045 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.972049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.972127 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.972210 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:24.972849 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.973848 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.975802 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.976524 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.976640 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.976752 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.977101 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.977222 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:02:24.977470 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.970926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47119 (local address 127.0.97.3:39329)
0620 07:02:24.971140 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:24.971148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:24.971774 (+ 626us) server_negotiation.cc:407] Connection header received
0620 07:02:24.971951 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:24.971956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:24.972033 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:24.972107 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 07:02:24.972729 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.973846 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.975776 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.976559 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.976676 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.976772 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:24.977115 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 07:02:24.977242 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:02:24.977797 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.970882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39329 (local address 127.0.97.1:47119)
0620 07:02:24.971665 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.971684 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.971812 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.972146 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.972151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.972173 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:24.972584 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.972596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.974034 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.974038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.975595 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.975608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.976762 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.976768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.976968 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.976999 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.977079 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.977203 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":29,"thread_start_us":441,"threads_started":1}
I20260620 07:02:24.977777 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:24.970997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40753 (local address 127.0.97.1:51437)
0620 07:02:24.971760 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:24.971777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:02:24.971916 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:02:24.972245 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:02:24.972249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:02:24.972280 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 07:02:24.972673 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.972684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.973964 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.973968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.975621 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:02:24.975636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:02:24.976704 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:02:24.976711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:02:24.976955 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:24.976980 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 07:02:24.977056 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 07:02:24.977178 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":111,"thread_start_us":480,"threads_started":1}
I20260620 07:02:24.978777 20534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc357379ad2142ceac6d3af20678b4ef" candidate_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0328c3256c4ec691e7167f811ec870" is_pre_election: true
I20260620 07:02:24.978786 20664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc357379ad2142ceac6d3af20678b4ef" candidate_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68def7d0cc2240718f3a49de1980c38b" is_pre_election: true
I20260620 07:02:24.979239 20534 raft_consensus.cc:2454] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebae3ecd2cd4278ab8d1aaaa9774548 in term 0.
I20260620 07:02:24.979241 20664 raft_consensus.cc:2454] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cebae3ecd2cd4278ab8d1aaaa9774548 in term 0.
I20260620 07:02:24.979835 20339 leader_election.cc:302] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68def7d0cc2240718f3a49de1980c38b, cebae3ecd2cd4278ab8d1aaaa9774548; no voters:
I20260620 07:02:24.980189 20734 raft_consensus.cc:2781] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 07:02:24.980321 20734 raft_consensus.cc:482] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 07:02:24.980460 20734 raft_consensus.cc:3037] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 0 FOLLOWER]: Advancing to term 1
W20260620 07:02:24.981750 20710 tablet.cc:2369] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:24.982779 20734 raft_consensus.cc:504] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.983451 20734 leader_election.cc:288] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [CANDIDATE]: Term 1 election: Requested vote from peers 3c0328c3256c4ec691e7167f811ec870 (127.0.97.2:40753), 68def7d0cc2240718f3a49de1980c38b (127.0.97.3:39329)
I20260620 07:02:24.983843 20534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc357379ad2142ceac6d3af20678b4ef" candidate_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0328c3256c4ec691e7167f811ec870"
I20260620 07:02:24.983932 20664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc357379ad2142ceac6d3af20678b4ef" candidate_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68def7d0cc2240718f3a49de1980c38b"
I20260620 07:02:24.984032 20534 raft_consensus.cc:3037] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:24.984126 20664 raft_consensus.cc:3037] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:24.986282 20664 raft_consensus.cc:2454] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebae3ecd2cd4278ab8d1aaaa9774548 in term 1.
I20260620 07:02:24.986285 20534 raft_consensus.cc:2454] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cebae3ecd2cd4278ab8d1aaaa9774548 in term 1.
I20260620 07:02:24.986696 20337 leader_election.cc:302] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0328c3256c4ec691e7167f811ec870, cebae3ecd2cd4278ab8d1aaaa9774548; no voters:
I20260620 07:02:24.986994 20734 raft_consensus.cc:2781] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:24.987493 20734 raft_consensus.cc:686] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 1 LEADER]: Becoming Leader. State: Replica: cebae3ecd2cd4278ab8d1aaaa9774548, State: Running, Role: LEADER
I20260620 07:02:24.987952 20734 consensus_queue.cc:237] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } }
I20260620 07:02:24.992220 20262 catalog_manager.cc:5495] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 reported cstate change: term changed from 0 to 1, leader changed from <none> to cebae3ecd2cd4278ab8d1aaaa9774548 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cebae3ecd2cd4278ab8d1aaaa9774548" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39647 } health_report { overall_health: HEALTHY } } }
W20260620 07:02:25.040266 20580 tablet.cc:2369] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:02:25.049284 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.044328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.0.97.1:39647)
0620 07:02:25.044551 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:25.044557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:25.044574 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:02:25.044748 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:25.044752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:25.044847 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:25.044964 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:02:25.045257 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.045958 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.047290 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.047733 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.047826 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:25.048211 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:02:25.048735 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:02:25.048838 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:25.048940 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 07:02:25.049009 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:02:25.057370 20534 raft_consensus.cc:1261] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 1 FOLLOWER]: Refusing update from remote peer cebae3ecd2cd4278ab8d1aaaa9774548: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:25.057370 20664 raft_consensus.cc:1261] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 1 FOLLOWER]: Refusing update from remote peer cebae3ecd2cd4278ab8d1aaaa9774548: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:02:25.058254 20734 consensus_queue.cc:1035] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68def7d0cc2240718f3a49de1980c38b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:25.058612 20738 consensus_queue.cc:1035] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0328c3256c4ec691e7167f811ec870" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:02:25.081351 20742 mvcc.cc:205] Tried to move back new op lower bound from 7298821918946074624 to 7298821918674624512. Current Snapshot: MvccSnapshot[applied={T|T < 7298821918946074624}]
I20260620 07:02:25.085510 20261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:25.085847 20261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 07:02:25.090070 20261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b0e5977c5074836a39285f81aca7efa: Sending DeleteTablet for 3 replicas of tablet cc357379ad2142ceac6d3af20678b4ef
I20260620 07:02:25.091421 20644 tablet_service.cc:1514] Processing DeleteTablet for tablet cc357379ad2142ceac6d3af20678b4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:25 UTC) from {username='slave'} at 127.0.0.1:54258
I20260620 07:02:25.091459 20514 tablet_service.cc:1514] Processing DeleteTablet for tablet cc357379ad2142ceac6d3af20678b4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:25 UTC) from {username='slave'} at 127.0.0.1:56326
I20260620 07:02:25.091745 20383 tablet_service.cc:1514] Processing DeleteTablet for tablet cc357379ad2142ceac6d3af20678b4ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 07:02:25 UTC) from {username='slave'} at 127.0.0.1:34702
I20260620 07:02:25.093523 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20321
I20260620 07:02:25.103804 20750 tablet_replica.cc:330] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: stopping tablet replica
I20260620 07:02:25.104483 20751 tablet_replica.cc:330] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: stopping tablet replica
I20260620 07:02:25.105216 20751 raft_consensus.cc:2229] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:25.105767 20750 raft_consensus.cc:2229] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 07:02:25.106446 20751 raft_consensus.cc:2258] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:25.107034 20750 raft_consensus.cc:2258] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:25.109545 20751 ts_tablet_manager.cc:1905] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:25.110981 20750 ts_tablet_manager.cc:1905] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 07:02:25.116122 20752 tablet_replica.cc:330] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548: stopping tablet replica
I20260620 07:02:25.116591 20752 raft_consensus.cc:2229] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 1 LEADER]: Raft consensus shutting down.
I20260620 07:02:25.117674 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.095291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34714 (local address 127.0.97.1:39647)
0620 07:02:25.095505 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:25.095512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:25.109344 (+ 13832us) server_negotiation.cc:407] Connection header received
0620 07:02:25.109845 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:25.109851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:25.109989 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:25.110086 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:25.110792 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.111932 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.113714 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.114381 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.114482 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:25.115869 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:25.115881 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:25.115899 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:25.116705 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:25.116771 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:25.116802 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:25.116925 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:25.117162 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 07:02:25.117276 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 07:02:25.118439 20443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:25.122130 20750 ts_tablet_manager.cc:1918] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:25.122308 20750 log.cc:1196] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-2/wal/wals/cc357379ad2142ceac6d3af20678b4ef
I20260620 07:02:25.122658 20751 ts_tablet_manager.cc:1918] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 07:02:25.122812 20751 log.cc:1196] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Deleting WAL directory at /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781938634772379-388-0/minicluster-data/ts-1/wal/wals/cc357379ad2142ceac6d3af20678b4ef
I20260620 07:02:25.122840 20750 ts_tablet_manager.cc:1939] T cc357379ad2142ceac6d3af20678b4ef P 68def7d0cc2240718f3a49de1980c38b: Deleting consensus metadata
I20260620 07:02:25.123224 20751 ts_tablet_manager.cc:1939] T cc357379ad2142ceac6d3af20678b4ef P 3c0328c3256c4ec691e7167f811ec870: Deleting consensus metadata
I20260620 07:02:25.124869 20249 catalog_manager.cc:4841] TS 68def7d0cc2240718f3a49de1980c38b (127.0.97.3:39329): tablet cc357379ad2142ceac6d3af20678b4ef (table test-table [id=2431d9e7362d481e8c15bf2e813b2a2c]) successfully deleted
I20260620 07:02:25.125244 20247 catalog_manager.cc:4841] TS 3c0328c3256c4ec691e7167f811ec870 (127.0.97.2:40753): tablet cc357379ad2142ceac6d3af20678b4ef (table test-table [id=2431d9e7362d481e8c15bf2e813b2a2c]) successfully deleted
I20260620 07:02:25.117290 20752 raft_consensus.cc:2258] T cc357379ad2142ceac6d3af20678b4ef P cebae3ecd2cd4278ab8d1aaaa9774548 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 07:02:25.215760 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20321
W20260620 07:02:25.231309 20246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39647 (error 108)
I20260620 07:02:25.231456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20452
I20260620 07:02:25.233027 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.232624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34724)
0620 07:02:25.232782 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.232892 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 07:02:25.234118 20246 catalog_manager.cc:4584] TS cebae3ecd2cd4278ab8d1aaaa9774548 (127.0.97.1:39647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc357379ad2142ceac6d3af20678b4ef: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
I20260620 07:02:25.237761 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.232215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.97.2:40753)
0620 07:02:25.232401 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:25.232407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:25.232427 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:02:25.232598 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:25.232604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:25.232705 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:25.232788 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:25.233176 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.233923 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.235154 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.235604 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.235701 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:25.236568 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:25.236576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:25.236588 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:25.237172 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:25.237217 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:25.237243 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:25.237336 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:25.237436 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 07:02:25.237519 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:02:25.238374 20574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:25.287776 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.287295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34726)
0620 07:02:25.287500 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.287589 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 07:02:25.288760 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.288412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34742)
0620 07:02:25.288559 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.288638 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 07:02:25.327184 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20452
I20260620 07:02:25.344310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20582
I20260620 07:02:25.349242 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.348778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34758)
0620 07:02:25.348982 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.349084 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 07:02:25.350251 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.349938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34760)
0620 07:02:25.350063 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.350133 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 07:02:25.351059 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.345118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54266 (local address 127.0.97.3:39329)
0620 07:02:25.345301 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:25.345306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:25.345325 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:02:25.345458 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:25.345463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:25.345548 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:25.345628 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:25.345972 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.346796 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.348110 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:25.348585 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:25.348711 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:25.349693 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:25.349701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:25.349716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:25.350395 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:25.350455 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:25.350476 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:25.350596 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:25.350695 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 07:02:25.350779 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 07:02:25.351634 20704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:25.431258 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.430673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34762)
0620 07:02:25.430918 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.431045 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 07:02:25.432394 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:25.431958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39647 (local address 127.0.0.1:34774)
0620 07:02:25.432159 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 07:02:25.432264 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 07:02:25.442446 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20582
I20260620 07:02:25.455417 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20231
I20260620 07:02:25.456449 20292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:25.564570 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20231
2026-06-20T07:02:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2733 ms)
[----------] 6 tests from EncryptionPolicyTest (20403 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:25 dist-test-slave-mz20 krb5kdc[20761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:25 dist-test-slave-mz20 krb5kdc[20761](info): set up 2 sockets
Jun 20 07:02:25 dist-test-slave-mz20 krb5kdc[20761](info): commencing operation
krb5kdc: starting...
W20260620 07:02:26.582760 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:26 dist-test-slave-mz20 krb5kdc[20761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:26.827530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:27.147981 20774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:27.148298 20774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:27.148423 20774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:27.155706 20774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:27.155836 20774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:27.155916 20774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:27.156008 20774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:27.156070 20774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:27.165931 20774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:27.167307 20774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:27.168635 20774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:27.180261 20779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:27.180229 20780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:27.180183 20782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:27.181263 20774 server_base.cc:1034] running on GCE node
I20260620 07:02:27.182057 20774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:27.183529 20774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:27.184737 20774 hybrid_clock.cc:648] HybridClock initialized: now 1781938947184695 us; error 34 us; skew 500 ppm
I20260620 07:02:27.185106 20774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:27 dist-test-slave-mz20 krb5kdc[20761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:27.188133 20774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:27.189637 20774 webserver.cc:460] Webserver started at http://127.0.0.1:38633/ using document root <none> and password file <none>
I20260620 07:02:27.190342 20774 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:27.190459 20774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:27.190786 20774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:27.193857 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "b7e6415264824692b815080bcadda276"
format_stamp: "Formatted at 2026-06-20 07:02:27 on dist-test-slave-mz20"
server_key: "d5b97ba4be7b9166f43d8514e31c7b7f"
server_key_iv: "99605c5cbed102bf7eab7ce1500b3a1a"
server_key_version: "encryptionkey@0"
I20260620 07:02:27.194677 20774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7e6415264824692b815080bcadda276"
format_stamp: "Formatted at 2026-06-20 07:02:27 on dist-test-slave-mz20"
server_key: "d5b97ba4be7b9166f43d8514e31c7b7f"
server_key_iv: "99605c5cbed102bf7eab7ce1500b3a1a"
server_key_version: "encryptionkey@0"
I20260620 07:02:27.200011 20774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 07:02:27.203639 20789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:27.205099 20774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:02:27.205279 20774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "b7e6415264824692b815080bcadda276"
format_stamp: "Formatted at 2026-06-20 07:02:27 on dist-test-slave-mz20"
server_key: "d5b97ba4be7b9166f43d8514e31c7b7f"
server_key_iv: "99605c5cbed102bf7eab7ce1500b3a1a"
server_key_version: "encryptionkey@0"
I20260620 07:02:27.205470 20774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:27.228725 20774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:27.233445 20774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:27.233733 20774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:27.251240 20774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35099
I20260620 07:02:27.251259 20840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35099 every 8 connection(s)
I20260620 07:02:27.252585 20774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:27.256269 20841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:27.261423 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20774
I20260620 07:02:27.261620 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:27.262238 388 external_mini_cluster.cc:1428] Setting key ff93518e9451bb4cde17af3ec9365155
I20260620 07:02:27.267092 20841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276: Bootstrap starting.
I20260620 07:02:27.270479 20841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:27.271618 20841 log.cc:824] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:27.275136 20841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276: No bootstrap required, opened a new log
Jun 20 07:02:27 dist-test-slave-mz20 krb5kdc[20761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:02:27.284402 20841 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } }
I20260620 07:02:27.284837 20841 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:27.284930 20841 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7e6415264824692b815080bcadda276, State: Initialized, Role: FOLLOWER
I20260620 07:02:27.285526 20841 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } }
I20260620 07:02:27.285738 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:27.264087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44227 (local address 127.0.0.1:35099)
0620 07:02:27.264876 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:27.264888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:27.264931 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:02:27.266706 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:27.266733 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:27.267136 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:27.267504 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:27.268982 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:27.270294 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:27.271709 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:27.272662 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:27.272853 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:27.277709 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:27.277720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:27.277738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:27.277787 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:27.281236 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:27.282276 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:27.282283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:27.282289 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:27.282482 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:27.283008 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:27.283012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:27.283014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:27.283569 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:27.283790 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:27.284272 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 07:02:27.284549 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":341,"threads_started":1}
I20260620 07:02:27.285761 20841 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:27.285954 20841 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:27.286144 20841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:27.289089 20841 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } }
I20260620 07:02:27.289652 20841 leader_election.cc:302] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7e6415264824692b815080bcadda276; no voters:
I20260620 07:02:27.289760 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:27.290221 20841 leader_election.cc:288] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:27.290369 20846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:27.292145 20841 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:27.299604 20846 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [term 1 LEADER]: Becoming Leader. State: Replica: b7e6415264824692b815080bcadda276, State: Running, Role: LEADER
I20260620 07:02:27.300613 20846 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } }
I20260620 07:02:27.303992 20847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7e6415264824692b815080bcadda276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } } }
I20260620 07:02:27.304311 20847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:27.305130 20865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:27.305666 20866 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:02:27.305505 20848 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7e6415264824692b815080bcadda276. Latest consensus state: current_term: 1 leader_uuid: "b7e6415264824692b815080bcadda276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7e6415264824692b815080bcadda276" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35099 } } }
I20260620 07:02:27.305927 20848 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:27.310796 20865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:27.317466 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:27.302321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38473 (local address 127.0.0.1:35099)
0620 07:02:27.302503 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:27.302510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:27.302859 (+ 349us) server_negotiation.cc:407] Connection header received
0620 07:02:27.303114 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:27.303119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:27.303198 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:27.303306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:27.305090 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:27.306384 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:27.308033 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:27.308517 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:27.308648 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:27.311307 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:27.311312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:27.311316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:27.311348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:27.314154 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:27.315108 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:27.315114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:27.315118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:27.315269 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:27.315784 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:27.315789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:27.315793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:27.316208 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:27.316481 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:27.316901 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 07:02:27.317092 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 07:02:27.320364 20865 catalog_manager.cc:1347] Generated new cluster ID: a077432d6b3c451bbbdb39e38ed926c1
I20260620 07:02:27.320473 20865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:27.343248 20865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:27.344633 20865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:27.360237 20865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7e6415264824692b815080bcadda276: Generated new TSK 0
I20260620 07:02:27.361158 20865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:27.374210 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20774
I20260620 07:02:27.376446 20836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:27.471560 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20774
W20260620 07:02:27.487084 20759 connection.cc:535] client connection to 127.0.0.1:35099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35099): Connection reset by peer (error 104)
2026-06-20T07:02:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:27 dist-test-slave-mz20 krb5kdc[20873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:27 dist-test-slave-mz20 krb5kdc[20873](info): set up 2 sockets
Jun 20 07:02:27 dist-test-slave-mz20 krb5kdc[20873](info): commencing operation
krb5kdc: starting...
W20260620 07:02:28.498634 388 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:28 dist-test-slave-mz20 krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:28.737437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:29.062485 20886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:29.062836 20886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:29.062983 20886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:29.070405 20886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:29.070572 20886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:29.070659 20886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:29.070739 20886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:29.070799 20886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:29.080977 20886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36017
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:29.082301 20886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:29.083586 20886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:29.095350 20894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:29.095304 20892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:29.095381 20891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:29.095817 20886 server_base.cc:1034] running on GCE node
I20260620 07:02:29.096750 20886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:29.098127 20886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:29.099330 20886 hybrid_clock.cc:648] HybridClock initialized: now 1781938949099276 us; error 39 us; skew 500 ppm
I20260620 07:02:29.099656 20886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:29 dist-test-slave-mz20 krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:29.102769 20886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:29.104242 20886 webserver.cc:460] Webserver started at http://127.0.0.1:35809/ using document root <none> and password file <none>
I20260620 07:02:29.104892 20886 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:29.105005 20886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:29.105281 20886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:29.108247 20886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "4e65c3acf26f4be1b7c1309255d807fc"
format_stamp: "Formatted at 2026-06-20 07:02:29 on dist-test-slave-mz20"
server_key: "443e3a42f1086eb72d04befc992845bf"
server_key_iv: "716d1692c1cff8fbae64f6a5b548b3cc"
server_key_version: "encryptionkey@0"
I20260620 07:02:29.108956 20886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "4e65c3acf26f4be1b7c1309255d807fc"
format_stamp: "Formatted at 2026-06-20 07:02:29 on dist-test-slave-mz20"
server_key: "443e3a42f1086eb72d04befc992845bf"
server_key_iv: "716d1692c1cff8fbae64f6a5b548b3cc"
server_key_version: "encryptionkey@0"
I20260620 07:02:29.114034 20886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 07:02:29.117524 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:29.118876 20886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:29.119033 20886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "4e65c3acf26f4be1b7c1309255d807fc"
format_stamp: "Formatted at 2026-06-20 07:02:29 on dist-test-slave-mz20"
server_key: "443e3a42f1086eb72d04befc992845bf"
server_key_iv: "716d1692c1cff8fbae64f6a5b548b3cc"
server_key_version: "encryptionkey@0"
I20260620 07:02:29.119216 20886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:29.139068 20886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:29.143263 20886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:29.143540 20886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:29.160900 20886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36017
I20260620 07:02:29.160943 20952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36017 every 8 connection(s)
I20260620 07:02:29.162233 20886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:29.165894 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:29.170816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20886
I20260620 07:02:29.171016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:29.171509 388 external_mini_cluster.cc:1428] Setting key 6e141068db22449d072e94d6b3026f95
I20260620 07:02:29.177013 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc: Bootstrap starting.
I20260620 07:02:29.180478 20953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:29.181665 20953 log.cc:824] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:29.184621 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:29.173242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41191 (local address 127.0.0.1:36017)
0620 07:02:29.173982 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:29.173993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:29.174035 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:02:29.175767 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:29.175788 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:29.176160 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:29.176459 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:29.177032 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:29.178295 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:29.180169 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:29.180889 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:29.181019 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:29.182325 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:29.182337 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:29.182359 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:29.183089 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:29.183168 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:29.183193 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:29.183309 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:29.183560 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 07:02:29.183723 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260620 07:02:29.184762 20953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc: No bootstrap required, opened a new log
I20260620 07:02:29.188035 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:29.194146 20953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } }
I20260620 07:02:29.194607 20953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:29.194702 20953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e65c3acf26f4be1b7c1309255d807fc, State: Initialized, Role: FOLLOWER
I20260620 07:02:29.195237 20953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } }
I20260620 07:02:29.195498 20953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:29.195637 20953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:29.195797 20953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:29.199107 20953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } }
I20260620 07:02:29.199738 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:29.193050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42685 (local address 127.0.0.1:36017)
0620 07:02:29.193189 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:29.193193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:29.193488 (+ 295us) server_negotiation.cc:407] Connection header received
0620 07:02:29.193666 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:29.193669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:29.193727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:29.193795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:29.194339 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:29.195249 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:29.197095 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:29.197570 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:29.197663 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:29.198912 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:29.198917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:29.198921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:29.198939 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:29.199001 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:29.199015 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:29.199099 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:29.199328 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 07:02:29.199382 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 07:02:29.199680 20953 leader_election.cc:302] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e65c3acf26f4be1b7c1309255d807fc; no voters:
I20260620 07:02:29.200335 20953 leader_election.cc:288] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:29.200441 20963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:29.201686 20963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [term 1 LEADER]: Becoming Leader. State: Replica: 4e65c3acf26f4be1b7c1309255d807fc, State: Running, Role: LEADER
I20260620 07:02:29.202178 20963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } }
I20260620 07:02:29.202306 20953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:29.204684 20965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e65c3acf26f4be1b7c1309255d807fc. Latest consensus state: current_term: 1 leader_uuid: "4e65c3acf26f4be1b7c1309255d807fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } } }
I20260620 07:02:29.205142 20965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:29.204701 20964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e65c3acf26f4be1b7c1309255d807fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e65c3acf26f4be1b7c1309255d807fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36017 } } }
I20260620 07:02:29.205549 20964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:29.206846 20969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:29.212136 20969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:29.221208 20969 catalog_manager.cc:1347] Generated new cluster ID: 47ac3e7a687f4bcd94303e62f12f9e16
I20260620 07:02:29.221338 20969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:29.244114 20969 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:29.246493 20969 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:29.269407 20969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e65c3acf26f4be1b7c1309255d807fc: Generated new TSK 0
I20260620 07:02:29.270512 20969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:29.286264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20886
W20260620 07:02:29.287468 20948 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41191
W20260620 07:02:29.288331 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:29.288504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20886
W20260620 07:02:29.302525 20871 connection.cc:535] client connection to 127.0.0.1:36017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36017): Connection reset by peer (error 104)
2026-06-20T07:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1815 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:29 dist-test-slave-mz20 krb5kdc[20984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:29 dist-test-slave-mz20 krb5kdc[20984](info): set up 2 sockets
Jun 20 07:02:29 dist-test-slave-mz20 krb5kdc[20984](info): commencing operation
krb5kdc: starting...
W20260620 07:02:30.315129 388 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:30 dist-test-slave-mz20 krb5kdc[20984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:30.613878 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:31.002794 20997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:31.003146 20997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:31.003286 20997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:31.011144 20997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:31.011288 20997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:31.011368 20997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:31.011447 20997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:31.011508 20997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:31.022504 20997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:31.024255 20997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:31.025951 20997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:31.039925 21002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:31.039814 21005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:31.039844 21003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:31.040489 20997 server_base.cc:1034] running on GCE node
I20260620 07:02:31.041536 20997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:31.043408 20997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:31.044617 20997 hybrid_clock.cc:648] HybridClock initialized: now 1781938951044565 us; error 44 us; skew 500 ppm
I20260620 07:02:31.045017 20997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:31 dist-test-slave-mz20 krb5kdc[20984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:31.048521 20997 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:31.050338 20997 webserver.cc:460] Webserver started at http://127.0.0.1:44127/ using document root <none> and password file <none>
I20260620 07:02:31.051199 20997 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:31.051321 20997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:31.051664 20997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:31.055210 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8"
format_stamp: "Formatted at 2026-06-20 07:02:31 on dist-test-slave-mz20"
server_key: "ffaa0cc7176f6e82cf29b9d9d3713ef9"
server_key_iv: "c61e6a8989a6e5e67a083009cfc6015a"
server_key_version: "encryptionkey@0"
I20260620 07:02:31.056035 20997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8"
format_stamp: "Formatted at 2026-06-20 07:02:31 on dist-test-slave-mz20"
server_key: "ffaa0cc7176f6e82cf29b9d9d3713ef9"
server_key_iv: "c61e6a8989a6e5e67a083009cfc6015a"
server_key_version: "encryptionkey@0"
I20260620 07:02:31.061843 20997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:31.065636 21012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:31.067139 20997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:31.067312 20997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8"
format_stamp: "Formatted at 2026-06-20 07:02:31 on dist-test-slave-mz20"
server_key: "ffaa0cc7176f6e82cf29b9d9d3713ef9"
server_key_iv: "c61e6a8989a6e5e67a083009cfc6015a"
server_key_version: "encryptionkey@0"
I20260620 07:02:31.067508 20997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:31.073899 20997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:31.079633 20997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:31.079972 20997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:31.098083 20997 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34817
I20260620 07:02:31.098106 21063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34817 every 8 connection(s)
I20260620 07:02:31.099684 20997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:31.104019 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:31.104573 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 20997
I20260620 07:02:31.104758 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:31.105283 388 external_mini_cluster.cc:1428] Setting key d58026ed3d4544a8e50393f3f95b14d3
I20260620 07:02:31.114185 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:31.107290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37049 (local address 127.0.0.1:34817)
0620 07:02:31.108207 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:31.108222 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:31.108270 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:02:31.110641 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:31.110675 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:31.111133 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:31.111528 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:31.112563 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:31.112570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:31.112583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:31.112597 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:31.112653 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:31.112676 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:31.112854 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:31.112982 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 07:02:31.113173 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":413,"threads_started":1}
I20260620 07:02:31.117862 21064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8: Bootstrap starting.
I20260620 07:02:31.118089 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:31.121978 21064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:31.123340 21064 log.cc:824] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:31.125571 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:31.123243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48005 (local address 127.0.0.1:34817)
0620 07:02:31.123436 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:31.123440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:31.123693 (+ 253us) server_negotiation.cc:407] Connection header received
0620 07:02:31.123873 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:31.123879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:31.123949 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:31.124066 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:31.124883 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:31.124888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:31.124893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:31.124912 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:31.124957 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:31.124977 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:31.125095 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:31.125239 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 07:02:31.125297 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 07:02:31.126849 21064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8: No bootstrap required, opened a new log
I20260620 07:02:31.134277 21064 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } }
I20260620 07:02:31.134757 21064 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:31.134850 21064 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc5f7cb2ab84cf8abc2f2de60615fa8, State: Initialized, Role: FOLLOWER
I20260620 07:02:31.135490 21064 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } }
I20260620 07:02:31.135732 21064 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:31.135865 21064 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:31.136031 21064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:31.138275 21064 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } }
I20260620 07:02:31.138975 21064 leader_election.cc:302] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc5f7cb2ab84cf8abc2f2de60615fa8; no voters:
I20260620 07:02:31.139576 21064 leader_election.cc:288] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:31.139706 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:31.140628 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [term 1 LEADER]: Becoming Leader. State: Replica: cdc5f7cb2ab84cf8abc2f2de60615fa8, State: Running, Role: LEADER
I20260620 07:02:31.141149 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } }
I20260620 07:02:31.141604 21064 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:31.144253 21076 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc5f7cb2ab84cf8abc2f2de60615fa8. Latest consensus state: current_term: 1 leader_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } } }
I20260620 07:02:31.144182 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5f7cb2ab84cf8abc2f2de60615fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34817 } } }
I20260620 07:02:31.144670 21076 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:31.144687 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:31.145571 21083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:31.152079 21083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:31.162298 21083 catalog_manager.cc:1347] Generated new cluster ID: 1bd90d0ea1274870b412ea16e525f35f
I20260620 07:02:31.162433 21083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:31.185748 21083 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:31.186182 21083 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:31.193759 21083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc5f7cb2ab84cf8abc2f2de60615fa8: Generated new TSK 0
I20260620 07:02:31.194661 21083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:31.198859 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20997
W20260620 07:02:31.199945 21059 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37049
W20260620 07:02:31.200610 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 20997: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:31.200745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 20997
2026-06-20T07:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:31 dist-test-slave-mz20 krb5kdc[21095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:31 dist-test-slave-mz20 krb5kdc[21095](info): set up 2 sockets
Jun 20 07:02:31 dist-test-slave-mz20 krb5kdc[21095](info): commencing operation
krb5kdc: starting...
W20260620 07:02:32.226831 388 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:32 dist-test-slave-mz20 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:32.467469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:32.810792 21108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:32.811118 21108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:32.811251 21108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:32.818930 21108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:32.819064 21108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:32.819144 21108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:32.819227 21108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:32.819288 21108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:32.829252 21108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:32.830646 21108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:32.832020 21108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:32.843430 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:32.843390 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:32.843459 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:32.844264 21108 server_base.cc:1034] running on GCE node
I20260620 07:02:32.845001 21108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:32.846454 21108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:32.847677 21108 hybrid_clock.cc:648] HybridClock initialized: now 1781938952847620 us; error 48 us; skew 500 ppm
I20260620 07:02:32.848007 21108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:32 dist-test-slave-mz20 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:32.853957 21108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:32.855523 21108 webserver.cc:460] Webserver started at http://127.0.0.1:42573/ using document root <none> and password file <none>
I20260620 07:02:32.856223 21108 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:32.856344 21108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:32.856652 21108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:32.859709 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "48fe795fe9c6418d97a40dbcb96e2821"
format_stamp: "Formatted at 2026-06-20 07:02:32 on dist-test-slave-mz20"
server_key: "a776e499b2a099705c5a69ebe341c9bb"
server_key_iv: "372a1bde3a03e0daabe58bb4fe1b56c3"
server_key_version: "encryptionkey@0"
I20260620 07:02:32.860545 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "48fe795fe9c6418d97a40dbcb96e2821"
format_stamp: "Formatted at 2026-06-20 07:02:32 on dist-test-slave-mz20"
server_key: "a776e499b2a099705c5a69ebe341c9bb"
server_key_iv: "372a1bde3a03e0daabe58bb4fe1b56c3"
server_key_version: "encryptionkey@0"
I20260620 07:02:32.865721 21108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 07:02:32.869297 21123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:32.870839 21108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:02:32.870997 21108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "48fe795fe9c6418d97a40dbcb96e2821"
format_stamp: "Formatted at 2026-06-20 07:02:32 on dist-test-slave-mz20"
server_key: "a776e499b2a099705c5a69ebe341c9bb"
server_key_iv: "372a1bde3a03e0daabe58bb4fe1b56c3"
server_key_version: "encryptionkey@0"
I20260620 07:02:32.871178 21108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:32.883852 21108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:32.887825 21108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:32.888064 21108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:32.904722 21108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39109
I20260620 07:02:32.904752 21174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39109 every 8 connection(s)
I20260620 07:02:32.906073 21108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:32.909848 21175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:32.911057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21108
I20260620 07:02:32.911242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:32.911763 388 external_mini_cluster.cc:1428] Setting key 8d5cceb3988ab35a767043c1c96be391
I20260620 07:02:32.922755 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821: Bootstrap starting.
Jun 20 07:02:32 dist-test-slave-mz20 krb5kdc[21095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:02:32.925858 21175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:32.926957 21175 log.cc:824] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:32.930131 21175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821: No bootstrap required, opened a new log
I20260620 07:02:32.933300 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:32.913425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44579 (local address 127.0.0.1:39109)
0620 07:02:32.914207 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:32.914223 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:32.914276 (+ 53us) server_negotiation.cc:407] Connection header received
0620 07:02:32.916392 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:32.916425 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:32.916962 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:32.917390 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:32.918668 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:32.919912 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:32.921128 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:32.921748 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:32.921903 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:32.925823 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:32.925831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:32.925848 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:32.925884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:32.929236 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:32.929896 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:32.929905 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:32.929912 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:32.930131 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:32.930616 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:32.930620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:32.930622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:32.931208 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:32.931439 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:32.931899 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 07:02:32.932277 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":298,"threads_started":1}
I20260620 07:02:32.937243 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:32.939654 21175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260620 07:02:32.940140 21175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:32.940249 21175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48fe795fe9c6418d97a40dbcb96e2821, State: Initialized, Role: FOLLOWER
I20260620 07:02:32.940912 21175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260620 07:02:32.941216 21175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:32.941367 21175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:32.941540 21175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:32.944684 21175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260620 07:02:32.945307 21175 leader_election.cc:302] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48fe795fe9c6418d97a40dbcb96e2821; no voters:
I20260620 07:02:32.945874 21175 leader_election.cc:288] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:32.946007 21185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:32.947105 21185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [term 1 LEADER]: Becoming Leader. State: Replica: 48fe795fe9c6418d97a40dbcb96e2821, State: Running, Role: LEADER
I20260620 07:02:32.947667 21185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } }
I20260620 07:02:32.948088 21175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:32.950817 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48fe795fe9c6418d97a40dbcb96e2821. Latest consensus state: current_term: 1 leader_uuid: "48fe795fe9c6418d97a40dbcb96e2821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } } }
I20260620 07:02:32.951205 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:32.950802 21186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48fe795fe9c6418d97a40dbcb96e2821" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48fe795fe9c6418d97a40dbcb96e2821" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39109 } } }
I20260620 07:02:32.951390 21186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:32.955176 21192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:32.960023 21178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:32.942649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55713 (local address 127.0.0.1:39109)
0620 07:02:32.942815 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:32.942819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:32.943112 (+ 293us) server_negotiation.cc:407] Connection header received
0620 07:02:32.943249 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:32.943253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:32.943319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:32.943404 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:32.945109 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:32.945972 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:32.947660 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:32.948335 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:32.948448 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:32.951818 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:32.951826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:32.951831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:32.951880 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:32.955843 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:32.956691 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:32.956697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:32.956701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:32.956872 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:32.957345 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:32.957351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:32.957355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:32.957852 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:32.958130 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:32.959344 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0620 07:02:32.959605 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 07:02:32.960176 21192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:32.970585 21192 catalog_manager.cc:1347] Generated new cluster ID: f9e9c274cd914f1e85051041c2aa71aa
I20260620 07:02:32.970713 21192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:33.017800 21192 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:33.019695 21192 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:33.031971 21192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48fe795fe9c6418d97a40dbcb96e2821: Generated new TSK 0
I20260620 07:02:33.033064 21192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:33.049865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21108
I20260620 07:02:33.051134 21170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:33.146755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21108
2026-06-20T07:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:33 dist-test-slave-mz20 krb5kdc[21207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:33 dist-test-slave-mz20 krb5kdc[21207](info): set up 2 sockets
Jun 20 07:02:33 dist-test-slave-mz20 krb5kdc[21207](info): commencing operation
krb5kdc: starting...
W20260620 07:02:34.166735 388 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:34 dist-test-slave-mz20 krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:34.412079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:34.741199 21220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:34.741529 21220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:34.741621 21220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:34.749161 21220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:34.749300 21220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:34.749351 21220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:34.749432 21220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:34.749492 21220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:34.759702 21220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:34.760957 21220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:34.762225 21220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:34.774107 21226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:34.774011 21228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:34.774173 21225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:34.774889 21220 server_base.cc:1034] running on GCE node
I20260620 07:02:34.775683 21220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:34.777340 21220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:34.778677 21220 hybrid_clock.cc:648] HybridClock initialized: now 1781938954778616 us; error 47 us; skew 500 ppm
I20260620 07:02:34.779058 21220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:34 dist-test-slave-mz20 krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:34.782848 21220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:34.784768 21220 webserver.cc:460] Webserver started at http://127.0.0.1:32833/ using document root <none> and password file <none>
I20260620 07:02:34.785862 21220 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:34.786010 21220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:34.786386 21220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:34.789587 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "ce22be231e814ecb815018e90c6b2e79"
format_stamp: "Formatted at 2026-06-20 07:02:34 on dist-test-slave-mz20"
server_key: "d73283a4620d7f04d2d706f092d9fb30"
server_key_iv: "09c42e8a7bab3658e91674bc8c186bcd"
server_key_version: "encryptionkey@0"
I20260620 07:02:34.790370 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "ce22be231e814ecb815018e90c6b2e79"
format_stamp: "Formatted at 2026-06-20 07:02:34 on dist-test-slave-mz20"
server_key: "d73283a4620d7f04d2d706f092d9fb30"
server_key_iv: "09c42e8a7bab3658e91674bc8c186bcd"
server_key_version: "encryptionkey@0"
I20260620 07:02:34.796092 21220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:02:34.799877 21235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:34.801345 21220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:34.801525 21220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "ce22be231e814ecb815018e90c6b2e79"
format_stamp: "Formatted at 2026-06-20 07:02:34 on dist-test-slave-mz20"
server_key: "d73283a4620d7f04d2d706f092d9fb30"
server_key_iv: "09c42e8a7bab3658e91674bc8c186bcd"
server_key_version: "encryptionkey@0"
I20260620 07:02:34.801721 21220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:34.820046 21220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:34.824715 21220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:34.825019 21220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:34.843312 21220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33059
I20260620 07:02:34.843345 21286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33059 every 8 connection(s)
I20260620 07:02:34.844717 21220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:34.846251 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21220
I20260620 07:02:34.846426 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:34.846990 388 external_mini_cluster.cc:1428] Setting key fd18a98e4827552ef8fd2cdab8f3d11a
I20260620 07:02:34.849460 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:34.861058 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:34.848913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54067 (local address 127.0.0.1:33059)
0620 07:02:34.849886 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:34.849899 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:34.849942 (+ 43us) server_negotiation.cc:407] Connection header received
0620 07:02:34.851689 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:34.851711 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:34.852127 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:34.852513 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:34.853147 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:34.854568 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:34.856417 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:34.857145 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:34.857276 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:34.858635 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:34.858648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:34.858666 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:34.859266 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:34.859346 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:34.859375 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:34.859486 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:34.859752 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:02:34.860047 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":357,"threads_started":1}
I20260620 07:02:34.862263 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79: Bootstrap starting.
I20260620 07:02:34.864964 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:34.865790 21287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:34.866807 21287 log.cc:824] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:34.869915 21287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79: No bootstrap required, opened a new log
I20260620 07:02:34.875758 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:34.869138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52903 (local address 127.0.0.1:33059)
0620 07:02:34.869306 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:34.869313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:34.869693 (+ 380us) server_negotiation.cc:407] Connection header received
0620 07:02:34.869865 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:34.869871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:34.869937 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:34.870028 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:34.870685 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:34.871392 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:34.873262 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:34.873729 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:34.873822 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:34.875029 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:34.875034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:34.875037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:34.875056 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:34.875098 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:34.875115 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:34.875192 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:34.875335 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 07:02:34.875415 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 07:02:34.879065 21287 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } }
I20260620 07:02:34.879426 21287 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:34.879504 21287 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce22be231e814ecb815018e90c6b2e79, State: Initialized, Role: FOLLOWER
I20260620 07:02:34.879990 21287 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } }
I20260620 07:02:34.880185 21287 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:34.880282 21287 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:34.880424 21287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:34.882947 21287 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } }
I20260620 07:02:34.883422 21287 leader_election.cc:302] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce22be231e814ecb815018e90c6b2e79; no voters:
I20260620 07:02:34.883977 21287 leader_election.cc:288] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:34.884099 21297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:34.884702 21297 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [term 1 LEADER]: Becoming Leader. State: Replica: ce22be231e814ecb815018e90c6b2e79, State: Running, Role: LEADER
I20260620 07:02:34.885210 21297 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } }
I20260620 07:02:34.885881 21287 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:34.887894 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce22be231e814ecb815018e90c6b2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } } }
I20260620 07:02:34.887868 21299 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce22be231e814ecb815018e90c6b2e79. Latest consensus state: current_term: 1 leader_uuid: "ce22be231e814ecb815018e90c6b2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce22be231e814ecb815018e90c6b2e79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33059 } } }
I20260620 07:02:34.888278 21299 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:34.888278 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:34.891466 21303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:34.896394 21303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:34.906677 21303 catalog_manager.cc:1347] Generated new cluster ID: 7ccf8c008e474642938f11ca2cf5f00d
I20260620 07:02:34.906797 21303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:34.919354 21303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:34.921386 21303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:34.937899 21303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce22be231e814ecb815018e90c6b2e79: Generated new TSK 0
I20260620 07:02:34.939211 21303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:34.946594 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21220
W20260620 07:02:34.947759 21282 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54067
W20260620 07:02:34.948551 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21220: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:34.948701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21220
2026-06-20T07:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1803 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:35 dist-test-slave-mz20 krb5kdc[21318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:35 dist-test-slave-mz20 krb5kdc[21318](info): set up 2 sockets
Jun 20 07:02:35 dist-test-slave-mz20 krb5kdc[21318](info): commencing operation
krb5kdc: starting...
W20260620 07:02:35.986687 388 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:36 dist-test-slave-mz20 krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:36.230464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:36.557032 21331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:36.557350 21331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:36.557474 21331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:36.565179 21331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:36.565320 21331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:36.565379 21331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:36.565459 21331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:36.565513 21331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:36.575647 21331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:36.576961 21331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:36.578295 21331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:36.590338 21336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:36.590349 21337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:36.590220 21339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:36.591814 21331 server_base.cc:1034] running on GCE node
I20260620 07:02:36.592703 21331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:36.594244 21331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:36.595500 21331 hybrid_clock.cc:648] HybridClock initialized: now 1781938956595429 us; error 64 us; skew 500 ppm
I20260620 07:02:36.596005 21331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:36 dist-test-slave-mz20 krb5kdc[21318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:36.599388 21331 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:36.601017 21331 webserver.cc:460] Webserver started at http://127.0.0.1:32857/ using document root <none> and password file <none>
I20260620 07:02:36.601722 21331 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:36.601850 21331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:36.602162 21331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:36.605286 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "3f7ebaad46464c508602216c5bce6c14"
format_stamp: "Formatted at 2026-06-20 07:02:36 on dist-test-slave-mz20"
server_key: "dfcdb482f6ea5bfafd35b3806ac4ae31"
server_key_iv: "87e874f38e4d192466d61ea58ccd29fa"
server_key_version: "encryptionkey@0"
I20260620 07:02:36.606037 21331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "3f7ebaad46464c508602216c5bce6c14"
format_stamp: "Formatted at 2026-06-20 07:02:36 on dist-test-slave-mz20"
server_key: "dfcdb482f6ea5bfafd35b3806ac4ae31"
server_key_iv: "87e874f38e4d192466d61ea58ccd29fa"
server_key_version: "encryptionkey@0"
I20260620 07:02:36.611375 21331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:02:36.615228 21346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:36.616681 21331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:36.616911 21331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "3f7ebaad46464c508602216c5bce6c14"
format_stamp: "Formatted at 2026-06-20 07:02:36 on dist-test-slave-mz20"
server_key: "dfcdb482f6ea5bfafd35b3806ac4ae31"
server_key_iv: "87e874f38e4d192466d61ea58ccd29fa"
server_key_version: "encryptionkey@0"
I20260620 07:02:36.617118 21331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:36.623075 21331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:36.627538 21331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:36.627830 21331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:36.645695 21331 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42837
I20260620 07:02:36.645717 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42837 every 8 connection(s)
I20260620 07:02:36.647159 21331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:36.650957 21398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:36.653928 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21331
I20260620 07:02:36.654102 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:36.654647 388 external_mini_cluster.cc:1428] Setting key f5e79ea8dcc071d0d71f99aa40ee841b
I20260620 07:02:36.662909 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:36.656757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50829 (local address 127.0.0.1:42837)
0620 07:02:36.657590 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:36.657603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:36.657647 (+ 44us) server_negotiation.cc:407] Connection header received
0620 07:02:36.659458 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:36.659480 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:36.659850 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:36.660198 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:36.661285 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:36.661292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:36.661304 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:36.661321 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:36.661386 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:36.661407 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:36.661597 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:36.661727 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 07:02:36.661913 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":373,"threads_started":1}
I20260620 07:02:36.663625 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14: Bootstrap starting.
I20260620 07:02:36.666437 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:36.666913 21398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:36.668118 21398 log.cc:824] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:36.671718 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14: No bootstrap required, opened a new log
I20260620 07:02:36.673729 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:36.671433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52853 (local address 127.0.0.1:42837)
0620 07:02:36.671615 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:36.671621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:36.671977 (+ 356us) server_negotiation.cc:407] Connection header received
0620 07:02:36.672110 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:36.672119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:36.672196 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:36.672314 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:36.673094 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:36.673099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:36.673104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:36.673121 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:36.673176 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:36.673198 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:36.673314 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:36.673354 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 07:02:36.673414 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:02:36.679891 21398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260620 07:02:36.680327 21398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:36.680428 21398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7ebaad46464c508602216c5bce6c14, State: Initialized, Role: FOLLOWER
I20260620 07:02:36.681039 21398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260620 07:02:36.681272 21398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:36.681407 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:36.681553 21398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:36.683787 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260620 07:02:36.684286 21398 leader_election.cc:302] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f7ebaad46464c508602216c5bce6c14; no voters:
I20260620 07:02:36.684844 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:36.684984 21408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:36.685971 21408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [term 1 LEADER]: Becoming Leader. State: Replica: 3f7ebaad46464c508602216c5bce6c14, State: Running, Role: LEADER
I20260620 07:02:36.686594 21408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } }
I20260620 07:02:36.686970 21398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:36.689424 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f7ebaad46464c508602216c5bce6c14. Latest consensus state: current_term: 1 leader_uuid: "3f7ebaad46464c508602216c5bce6c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } } }
I20260620 07:02:36.689292 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f7ebaad46464c508602216c5bce6c14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7ebaad46464c508602216c5bce6c14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42837 } } }
I20260620 07:02:36.689731 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:36.689736 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:36.690619 21416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:36.696092 21416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:36.705819 21416 catalog_manager.cc:1347] Generated new cluster ID: c2ece3e89a914178bccc6a826ab173ad
I20260620 07:02:36.705940 21416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:36.723192 21416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:36.723623 21416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:36.732395 21416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f7ebaad46464c508602216c5bce6c14: Generated new TSK 0
I20260620 07:02:36.733229 21416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:36.744153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21331
W20260620 07:02:36.745427 21393 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50829
W20260620 07:02:36.746239 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21331: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:36.746376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21331
2026-06-20T07:02:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:36 dist-test-slave-mz20 krb5kdc[21429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:36 dist-test-slave-mz20 krb5kdc[21429](info): set up 2 sockets
Jun 20 07:02:36 dist-test-slave-mz20 krb5kdc[21429](info): commencing operation
krb5kdc: starting...
W20260620 07:02:37.778759 388 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:37 dist-test-slave-mz20 krb5kdc[21429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:38.023926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35763
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:38.346058 21442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:38.346370 21442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:38.346490 21442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:38.353885 21442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:38.354028 21442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:38.354080 21442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:38.354168 21442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:38.354224 21442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:38.364328 21442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:38.365581 21442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:38.366864 21442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:38.377838 21450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:38.377884 21448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:38.378371 21447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:38.379124 21442 server_base.cc:1034] running on GCE node
I20260620 07:02:38.379835 21442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:38.381145 21442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:38.382344 21442 hybrid_clock.cc:648] HybridClock initialized: now 1781938958382289 us; error 38 us; skew 500 ppm
I20260620 07:02:38.382709 21442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:38 dist-test-slave-mz20 krb5kdc[21429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:38.390126 21442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:38.391604 21442 webserver.cc:460] Webserver started at http://127.0.0.1:35421/ using document root <none> and password file <none>
I20260620 07:02:38.392256 21442 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:38.392383 21442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:38.392663 21442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:38.395638 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "b8b0978e427d431494a1cb145962fe00"
format_stamp: "Formatted at 2026-06-20 07:02:38 on dist-test-slave-mz20"
server_key: "d8e14952f955bb07f680d5338cf06a78"
server_key_iv: "36e9c3bc2027488125e4499177a3c193"
server_key_version: "encryptionkey@0"
I20260620 07:02:38.396390 21442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "b8b0978e427d431494a1cb145962fe00"
format_stamp: "Formatted at 2026-06-20 07:02:38 on dist-test-slave-mz20"
server_key: "d8e14952f955bb07f680d5338cf06a78"
server_key_iv: "36e9c3bc2027488125e4499177a3c193"
server_key_version: "encryptionkey@0"
I20260620 07:02:38.401342 21442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:38.404752 21457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:38.406164 21442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 07:02:38.406369 21442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "b8b0978e427d431494a1cb145962fe00"
format_stamp: "Formatted at 2026-06-20 07:02:38 on dist-test-slave-mz20"
server_key: "d8e14952f955bb07f680d5338cf06a78"
server_key_iv: "36e9c3bc2027488125e4499177a3c193"
server_key_version: "encryptionkey@0"
I20260620 07:02:38.406572 21442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:38.418464 21442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:38.422413 21442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:38.422694 21442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:38.439783 21442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35763
I20260620 07:02:38.439834 21508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35763 every 8 connection(s)
I20260620 07:02:38.441110 21442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:38.444669 21509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:38.447594 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21442
I20260620 07:02:38.447772 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:38.448205 388 external_mini_cluster.cc:1428] Setting key f2cb6378d37f912ddcaaff19a6da4052
I20260620 07:02:38.457240 21509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: Bootstrap starting.
I20260620 07:02:38.460839 21509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:38.461817 21509 log.cc:824] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:02:38 dist-test-slave-mz20 krb5kdc[21429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:02:38.464905 21509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: No bootstrap required, opened a new log
I20260620 07:02:38.471585 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:38.450012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:49937 (local address 127.0.0.1:35763)
0620 07:02:38.450764 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:38.450779 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:38.450828 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:02:38.452982 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:38.453010 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:38.453543 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:38.453958 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:38.455194 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:38.457076 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:38.458466 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:38.459530 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:38.459692 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:38.464170 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:38.464178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:38.464193 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:38.464243 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:38.467363 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:38.468325 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:38.468334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:38.468341 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:38.468512 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:38.468977 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:38.468980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:38.468982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:38.469511 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:38.469753 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:38.470200 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 07:02:38.470495 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":283,"threads_started":1}
I20260620 07:02:38.473846 21509 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } }
I20260620 07:02:38.474264 21509 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:38.474375 21509 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b0978e427d431494a1cb145962fe00, State: Initialized, Role: FOLLOWER
I20260620 07:02:38.474992 21509 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } }
I20260620 07:02:38.475268 21509 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:38.475423 21509 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:38.475595 21509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:38.475644 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:38.477655 21509 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } }
I20260620 07:02:38.478169 21509 leader_election.cc:302] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b0978e427d431494a1cb145962fe00; no voters:
I20260620 07:02:38.478829 21509 leader_election.cc:288] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:38.478957 21517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:38.482725 21517 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [term 1 LEADER]: Becoming Leader. State: Replica: b8b0978e427d431494a1cb145962fe00, State: Running, Role: LEADER
I20260620 07:02:38.483321 21517 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } }
I20260620 07:02:38.484706 21509 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:38.486275 21520 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b0978e427d431494a1cb145962fe00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } } }
I20260620 07:02:38.486663 21520 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:38.487577 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b0978e427d431494a1cb145962fe00. Latest consensus state: current_term: 1 leader_uuid: "b8b0978e427d431494a1cb145962fe00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b0978e427d431494a1cb145962fe00" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35763 } } }
I20260620 07:02:38.487910 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00 [sys.catalog]: This master's current role is: LEADER
W20260620 07:02:38.494729 21535 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:02:38.494848 21535 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:02:38.495012 21527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:38.496980 21527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:38.505762 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:38.480033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:51053 (local address 127.0.0.1:35763)
0620 07:02:38.480207 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:38.480224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:38.483866 (+ 3642us) server_negotiation.cc:407] Connection header received
0620 07:02:38.484032 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:38.484037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:38.484117 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:38.484212 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:38.491889 (+ 7677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:38.493140 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:38.494994 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:38.495735 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:38.495881 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:38.499013 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:38.499020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:38.499025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:38.499065 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:38.502804 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:38.503716 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:38.503720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:38.503722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:38.503846 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:38.504286 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:38.504290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:38.504292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:38.504585 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:38.504835 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:38.505193 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 07:02:38.505405 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:02:38.508872 21527 catalog_manager.cc:1347] Generated new cluster ID: 5ade70524e384521a7cdc906b0591cac
I20260620 07:02:38.508982 21527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:38.519309 21527 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:38.521133 21527 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:38.550120 21527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b0978e427d431494a1cb145962fe00: Generated new TSK 0
I20260620 07:02:38.551028 21527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:38.562842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21442
I20260620 07:02:38.563957 21504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:38.656994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21442
W20260620 07:02:38.671456 21427 connection.cc:535] client connection to 127.0.0.1:35763 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35763): Connection reset by peer (error 104)
2026-06-20T07:02:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:38 dist-test-slave-mz20 krb5kdc[21541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:38 dist-test-slave-mz20 krb5kdc[21541](info): set up 2 sockets
Jun 20 07:02:38 dist-test-slave-mz20 krb5kdc[21541](info): commencing operation
krb5kdc: starting...
W20260620 07:02:39.674641 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:39 dist-test-slave-mz20 krb5kdc[21541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:39.922222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:40.258127 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:40.258421 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:40.258522 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:40.266088 21554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:40.266203 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:40.266269 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:40.266350 21554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:40.266407 21554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:40.276531 21554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:40.277833 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:40.279147 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:40.290905 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:40.290983 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:40.290977 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:40.292156 21554 server_base.cc:1034] running on GCE node
I20260620 07:02:40.293071 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:40.294636 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:40.295892 21554 hybrid_clock.cc:648] HybridClock initialized: now 1781938960295822 us; error 54 us; skew 500 ppm
I20260620 07:02:40.296289 21554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:40 dist-test-slave-mz20 krb5kdc[21541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:40.299540 21554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:40.301172 21554 webserver.cc:460] Webserver started at http://127.0.0.1:46387/ using document root <none> and password file <none>
I20260620 07:02:40.301841 21554 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:40.301959 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:40.302279 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:40.305285 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "96d9d6be33164537898da8dd26ecb313"
format_stamp: "Formatted at 2026-06-20 07:02:40 on dist-test-slave-mz20"
server_key: "c21c2604bbc0cd45970668451a69c3ef"
server_key_iv: "c28ad3d6cdf6e3608c8463dd1129862f"
server_key_version: "encryptionkey@0"
I20260620 07:02:40.306057 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "96d9d6be33164537898da8dd26ecb313"
format_stamp: "Formatted at 2026-06-20 07:02:40 on dist-test-slave-mz20"
server_key: "c21c2604bbc0cd45970668451a69c3ef"
server_key_iv: "c28ad3d6cdf6e3608c8463dd1129862f"
server_key_version: "encryptionkey@0"
I20260620 07:02:40.311414 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 07:02:40.314945 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:40.316280 21554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:02:40.316439 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "96d9d6be33164537898da8dd26ecb313"
format_stamp: "Formatted at 2026-06-20 07:02:40 on dist-test-slave-mz20"
server_key: "c21c2604bbc0cd45970668451a69c3ef"
server_key_iv: "c28ad3d6cdf6e3608c8463dd1129862f"
server_key_version: "encryptionkey@0"
I20260620 07:02:40.316627 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:40.349004 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:40.353643 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:40.353945 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:40.373543 21554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35623
I20260620 07:02:40.373590 21620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35623 every 8 connection(s)
I20260620 07:02:40.374939 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:40.376927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21554
I20260620 07:02:40.377111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:40.377643 388 external_mini_cluster.cc:1428] Setting key e8360c2e91eae76fbd2c426f3043e9c5
I20260620 07:02:40.379491 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:40.391585 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:40.379630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:42753 (local address 127.0.0.1:35623)
0620 07:02:40.380425 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:40.380438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:40.380478 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:02:40.382196 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:40.382220 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:40.382707 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:40.383101 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:40.383664 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:40.385015 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:40.386327 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:40.387025 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:40.387196 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:40.388475 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:40.388482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:40.388499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:40.389208 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:40.389302 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:40.389344 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:40.389505 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:40.390029 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 07:02:40.390260 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":335,"threads_started":1}
I20260620 07:02:40.391863 21621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313: Bootstrap starting.
I20260620 07:02:40.395732 21621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:40.396293 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:40.396977 21621 log.cc:824] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:40.400458 21621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313: No bootstrap required, opened a new log
I20260620 07:02:40.408015 21624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:40.401198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:40047 (local address 127.0.0.1:35623)
0620 07:02:40.401373 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:40.401378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:40.401696 (+ 318us) server_negotiation.cc:407] Connection header received
0620 07:02:40.401844 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:40.401849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:40.401930 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:40.402048 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:40.402501 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:40.403684 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:40.405081 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:40.405846 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:40.405979 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:40.407201 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:40.407211 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:40.407216 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:40.407241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:40.407302 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:40.407322 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:40.407430 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:40.407630 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 07:02:40.407689 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:02:40.409694 21621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260620 07:02:40.410049 21621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:40.410140 21621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96d9d6be33164537898da8dd26ecb313, State: Initialized, Role: FOLLOWER
I20260620 07:02:40.410750 21621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260620 07:02:40.410984 21621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:40.411128 21621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:40.411331 21621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:40.413489 21621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260620 07:02:40.414000 21621 leader_election.cc:302] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96d9d6be33164537898da8dd26ecb313; no voters:
I20260620 07:02:40.414578 21621 leader_election.cc:288] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:40.414741 21631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:40.415345 21631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [term 1 LEADER]: Becoming Leader. State: Replica: 96d9d6be33164537898da8dd26ecb313, State: Running, Role: LEADER
I20260620 07:02:40.415838 21631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } }
I20260620 07:02:40.416510 21621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:40.418813 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96d9d6be33164537898da8dd26ecb313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } } }
I20260620 07:02:40.419241 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:40.419628 21633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96d9d6be33164537898da8dd26ecb313. Latest consensus state: current_term: 1 leader_uuid: "96d9d6be33164537898da8dd26ecb313" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96d9d6be33164537898da8dd26ecb313" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35623 } } }
I20260620 07:02:40.419945 21633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:40.420250 21639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:40.426070 21639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:40.435209 21639 catalog_manager.cc:1347] Generated new cluster ID: 373a922417ea4396923c24a8b0f1794d
I20260620 07:02:40.435321 21639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:40.468708 21639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:40.470018 21639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:40.477470 21639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96d9d6be33164537898da8dd26ecb313: Generated new TSK 0
I20260620 07:02:40.478259 21639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:40.489634 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21554
W20260620 07:02:40.490901 21616 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:42753
W20260620 07:02:40.491623 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21554: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:40.491745 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21554
W20260620 07:02:40.505710 21539 connection.cc:535] client connection to 127.0.0.1:35623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35623): Connection reset by peer (error 104)
2026-06-20T07:02:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:40 dist-test-slave-mz20 krb5kdc[21652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:40 dist-test-slave-mz20 krb5kdc[21652](info): set up 2 sockets
Jun 20 07:02:40 dist-test-slave-mz20 krb5kdc[21652](info): commencing operation
krb5kdc: starting...
W20260620 07:02:41.518755 388 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:41 dist-test-slave-mz20 krb5kdc[21652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:41.775444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:42.095587 21665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:42.095872 21665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:42.096006 21665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:42.103327 21665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:42.103437 21665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:42.103495 21665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:42.103581 21665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:42.103641 21665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:42.113541 21665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:42.114830 21665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:42.116103 21665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:42.127655 21671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:42.127724 21670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:42.127693 21673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:42.128671 21665 server_base.cc:1034] running on GCE node
I20260620 07:02:42.129478 21665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:42.130929 21665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:42.132140 21665 hybrid_clock.cc:648] HybridClock initialized: now 1781938962132092 us; error 34 us; skew 500 ppm
I20260620 07:02:42.132491 21665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:42 dist-test-slave-mz20 krb5kdc[21652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938962, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:42.135718 21665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:42.137306 21665 webserver.cc:460] Webserver started at http://127.0.0.1:41281/ using document root <none> and password file <none>
I20260620 07:02:42.137974 21665 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:42.138140 21665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:42.138432 21665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:42.141520 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "dd4179e29d314c66b54e0b7ed0f6bccb"
format_stamp: "Formatted at 2026-06-20 07:02:42 on dist-test-slave-mz20"
server_key: "dfed19ac33373e36a61c178f0a0f7ce2"
server_key_iv: "f49e46dfae89f884e70d6a67042d0f8f"
server_key_version: "encryptionkey@0"
I20260620 07:02:42.142310 21665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "dd4179e29d314c66b54e0b7ed0f6bccb"
format_stamp: "Formatted at 2026-06-20 07:02:42 on dist-test-slave-mz20"
server_key: "dfed19ac33373e36a61c178f0a0f7ce2"
server_key_iv: "f49e46dfae89f884e70d6a67042d0f8f"
server_key_version: "encryptionkey@0"
I20260620 07:02:42.147503 21665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 07:02:42.151129 21680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:42.152486 21665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 07:02:42.152643 21665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "dd4179e29d314c66b54e0b7ed0f6bccb"
format_stamp: "Formatted at 2026-06-20 07:02:42 on dist-test-slave-mz20"
server_key: "dfed19ac33373e36a61c178f0a0f7ce2"
server_key_iv: "f49e46dfae89f884e70d6a67042d0f8f"
server_key_version: "encryptionkey@0"
I20260620 07:02:42.152827 21665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:42.158473 21665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:42.162418 21665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:42.162714 21665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:42.179394 21665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42991
I20260620 07:02:42.179445 21731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42991 every 8 connection(s)
I20260620 07:02:42.180685 21665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:42.184168 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:42.188264 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21665
I20260620 07:02:42.188447 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:42.188987 388 external_mini_cluster.cc:1428] Setting key f5c73386191d141c8c363da5202556c8
I20260620 07:02:42.196625 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:42.190785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:39549 (local address 127.0.0.1:42991)
0620 07:02:42.191518 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:42.191529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:42.191569 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:02:42.193142 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:42.193165 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:42.193518 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:42.193859 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:42.194973 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:42.194983 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:42.194999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:42.195017 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:42.195086 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:42.195109 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:42.195298 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:42.195403 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 07:02:42.195593 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":323,"threads_started":1}
I20260620 07:02:42.197141 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb: Bootstrap starting.
I20260620 07:02:42.200739 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:42.200846 21732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:42.202080 21732 log.cc:824] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:42.205363 21732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb: No bootstrap required, opened a new log
I20260620 07:02:42.207886 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:42.205617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:42051 (local address 127.0.0.1:42991)
0620 07:02:42.205836 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:42.205845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:42.206091 (+ 246us) server_negotiation.cc:407] Connection header received
0620 07:02:42.206256 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:42.206259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:42.206308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:42.206383 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:42.207293 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:42.207297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:42.207301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:42.207318 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:42.207358 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:42.207375 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:42.207465 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:42.207573 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 07:02:42.207622 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 07:02:42.213639 21732 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260620 07:02:42.213990 21732 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:42.214082 21732 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4179e29d314c66b54e0b7ed0f6bccb, State: Initialized, Role: FOLLOWER
I20260620 07:02:42.214654 21732 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260620 07:02:42.214874 21732 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:42.215004 21732 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:42.215139 21732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:42.217597 21732 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260620 07:02:42.218062 21732 leader_election.cc:302] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd4179e29d314c66b54e0b7ed0f6bccb; no voters:
I20260620 07:02:42.218603 21732 leader_election.cc:288] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:42.218725 21742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:42.219329 21742 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [term 1 LEADER]: Becoming Leader. State: Replica: dd4179e29d314c66b54e0b7ed0f6bccb, State: Running, Role: LEADER
I20260620 07:02:42.219800 21742 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } }
I20260620 07:02:42.220547 21732 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:42.222378 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } } }
I20260620 07:02:42.222740 21744 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd4179e29d314c66b54e0b7ed0f6bccb. Latest consensus state: current_term: 1 leader_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd4179e29d314c66b54e0b7ed0f6bccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42991 } } }
I20260620 07:02:42.222915 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:42.222991 21744 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:42.225049 21748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:42.230406 21748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:42.239347 21748 catalog_manager.cc:1347] Generated new cluster ID: ec0ff97921054b479e4a5875a096472d
I20260620 07:02:42.239447 21748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:42.253087 21748 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:42.253540 21748 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:42.261375 21748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd4179e29d314c66b54e0b7ed0f6bccb: Generated new TSK 0
I20260620 07:02:42.262171 21748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:42.270140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21665
W20260620 07:02:42.271173 21727 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:39549
W20260620 07:02:42.271910 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21665: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:42.272085 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21665
2026-06-20T07:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:42 dist-test-slave-mz20 krb5kdc[21763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:42 dist-test-slave-mz20 krb5kdc[21763](info): set up 2 sockets
Jun 20 07:02:42 dist-test-slave-mz20 krb5kdc[21763](info): commencing operation
krb5kdc: starting...
W20260620 07:02:43.307121 388 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:43 dist-test-slave-mz20 krb5kdc[21763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:43.553650 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:43.876168 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:43.876475 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:43.876616 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:43.883706 21776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:43.883826 21776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:43.883903 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:43.883985 21776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:43.884043 21776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:43.894016 21776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:43.895361 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:43.896660 21776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:43.908056 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:43.908133 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:43.908226 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:43.909183 21776 server_base.cc:1034] running on GCE node
I20260620 07:02:43.909934 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:43.911407 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:43.912613 21776 hybrid_clock.cc:648] HybridClock initialized: now 1781938963912565 us; error 37 us; skew 500 ppm
I20260620 07:02:43.912930 21776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:43 dist-test-slave-mz20 krb5kdc[21763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:43.919061 21776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:43.920518 21776 webserver.cc:460] Webserver started at http://127.0.0.1:34927/ using document root <none> and password file <none>
I20260620 07:02:43.921154 21776 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:43.921252 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:43.921504 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:43.924425 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "c7d70fc18b074e269f69f4add077d136"
format_stamp: "Formatted at 2026-06-20 07:02:43 on dist-test-slave-mz20"
server_key: "d7e6226c7d3250a06b7f40c0edc0d85d"
server_key_iv: "b36329be57de88ce43e852d2465a5008"
server_key_version: "encryptionkey@0"
I20260620 07:02:43.925113 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "c7d70fc18b074e269f69f4add077d136"
format_stamp: "Formatted at 2026-06-20 07:02:43 on dist-test-slave-mz20"
server_key: "d7e6226c7d3250a06b7f40c0edc0d85d"
server_key_iv: "b36329be57de88ce43e852d2465a5008"
server_key_version: "encryptionkey@0"
I20260620 07:02:43.930063 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:43.933557 21791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:43.934864 21776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:02:43.935022 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "c7d70fc18b074e269f69f4add077d136"
format_stamp: "Formatted at 2026-06-20 07:02:43 on dist-test-slave-mz20"
server_key: "d7e6226c7d3250a06b7f40c0edc0d85d"
server_key_iv: "b36329be57de88ce43e852d2465a5008"
server_key_version: "encryptionkey@0"
I20260620 07:02:43.935194 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:43.961433 21776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:43.965358 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:43.965606 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:43.982646 21776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46469
I20260620 07:02:43.982664 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46469 every 8 connection(s)
I20260620 07:02:43.983970 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:43.987579 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:43.989460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21776
I20260620 07:02:43.989640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:43.990139 388 external_mini_cluster.cc:1428] Setting key fdcc084657187a8a41556aeac7eaf277
I20260620 07:02:44.001370 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136: Bootstrap starting.
Jun 20 07:02:44 dist-test-slave-mz20 krb5kdc[21763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 07:02:44.005064 21843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:44.006173 21843 log.cc:824] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:44.009143 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136: No bootstrap required, opened a new log
I20260620 07:02:44.013109 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:43.991959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:35973 (local address 127.0.0.1:46469)
0620 07:02:43.992780 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:43.992791 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:43.992830 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:02:43.994397 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:43.994418 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:43.994881 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:43.995189 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:43.996834 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:43.998142 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:43.999939 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:44.000616 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:44.000770 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:44.005584 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:44.005592 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:44.005606 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:44.005655 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:44.008752 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:44.009728 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:44.009735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:44.009740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:44.009924 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:44.010429 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:44.010432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:44.010434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:44.011013 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:44.011298 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:44.011734 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 07:02:44.012029 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":388,"threads_started":1}
I20260620 07:02:44.016865 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:44.017791 21843 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260620 07:02:44.018220 21843 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:44.018314 21843 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d70fc18b074e269f69f4add077d136, State: Initialized, Role: FOLLOWER
I20260620 07:02:44.018950 21843 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260620 07:02:44.019235 21843 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:44.019376 21843 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:44.019546 21843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:44.022521 21843 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260620 07:02:44.023185 21843 leader_election.cc:302] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7d70fc18b074e269f69f4add077d136; no voters:
I20260620 07:02:44.023861 21843 leader_election.cc:288] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:44.023900 21853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:44.026124 21843 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:44.026862 21853 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [term 1 LEADER]: Becoming Leader. State: Replica: c7d70fc18b074e269f69f4add077d136, State: Running, Role: LEADER
I20260620 07:02:44.027415 21853 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } }
I20260620 07:02:44.030937 21854 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7d70fc18b074e269f69f4add077d136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } } }
I20260620 07:02:44.030951 21855 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7d70fc18b074e269f69f4add077d136. Latest consensus state: current_term: 1 leader_uuid: "c7d70fc18b074e269f69f4add077d136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7d70fc18b074e269f69f4add077d136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46469 } } }
I20260620 07:02:44.031386 21855 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:44.031386 21854 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:44.032235 21861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:44.041133 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:44.021875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:41567 (local address 127.0.0.1:46469)
0620 07:02:44.022054 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:44.022060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:44.022442 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:02:44.022611 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:44.022616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:44.022700 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:44.022807 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:44.024770 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:44.025883 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:44.027740 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:44.028477 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:44.028589 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:44.032207 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:44.032215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:44.032225 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:44.032279 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:44.036111 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:44.037279 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:44.037286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:44.037291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:44.037461 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:44.038071 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:44.038076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:44.038080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:44.038702 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:44.039025 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:44.040481 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0620 07:02:44.040702 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:02:44.044113 21861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:44.052809 21861 catalog_manager.cc:1347] Generated new cluster ID: fa324a1c78fb43b4a978546a458e762b
I20260620 07:02:44.052929 21861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:44.063545 21861 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:44.064810 21861 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:44.071485 21861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7d70fc18b074e269f69f4add077d136: Generated new TSK 0
I20260620 07:02:44.072407 21861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:44.078490 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21776
I20260620 07:02:44.080055 21838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:44.169077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21776
W20260620 07:02:44.183151 21761 connection.cc:535] client connection to 127.0.0.1:46469 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46469): Connection reset by peer (error 104)
2026-06-20T07:02:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:44 dist-test-slave-mz20 krb5kdc[21875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:44 dist-test-slave-mz20 krb5kdc[21875](info): set up 2 sockets
Jun 20 07:02:44 dist-test-slave-mz20 krb5kdc[21875](info): commencing operation
krb5kdc: starting...
W20260620 07:02:45.178630 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:45 dist-test-slave-mz20 krb5kdc[21875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:45.427345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:45.751595 21888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:45.751883 21888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:45.752019 21888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:45.759351 21888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:45.759462 21888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:45.759533 21888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:45.759609 21888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:45.759667 21888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:45.769701 21888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:45.771090 21888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:45.772428 21888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:45.783593 21896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:45.783535 21893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:45.783560 21894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:45.784524 21888 server_base.cc:1034] running on GCE node
I20260620 07:02:45.785305 21888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:45.786763 21888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:45.787989 21888 hybrid_clock.cc:648] HybridClock initialized: now 1781938965787940 us; error 35 us; skew 500 ppm
I20260620 07:02:45.788321 21888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:45 dist-test-slave-mz20 krb5kdc[21875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:45.791508 21888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:45.793030 21888 webserver.cc:460] Webserver started at http://127.0.0.1:46179/ using document root <none> and password file <none>
I20260620 07:02:45.793690 21888 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:45.793803 21888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:45.794077 21888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:45.797245 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d"
format_stamp: "Formatted at 2026-06-20 07:02:45 on dist-test-slave-mz20"
server_key: "81236f5a952030c515bdc629fdd4bc31"
server_key_iv: "2267b90177fe4b04bcc00f39e482da49"
server_key_version: "encryptionkey@0"
I20260620 07:02:45.798029 21888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d"
format_stamp: "Formatted at 2026-06-20 07:02:45 on dist-test-slave-mz20"
server_key: "81236f5a952030c515bdc629fdd4bc31"
server_key_iv: "2267b90177fe4b04bcc00f39e482da49"
server_key_version: "encryptionkey@0"
I20260620 07:02:45.803002 21888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:02:45.806350 21903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:45.807690 21888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:45.807847 21888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d"
format_stamp: "Formatted at 2026-06-20 07:02:45 on dist-test-slave-mz20"
server_key: "81236f5a952030c515bdc629fdd4bc31"
server_key_iv: "2267b90177fe4b04bcc00f39e482da49"
server_key_version: "encryptionkey@0"
I20260620 07:02:45.808032 21888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:45.840036 21888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:45.844277 21888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:45.844551 21888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:45.861828 21888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39845
I20260620 07:02:45.861855 21954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39845 every 8 connection(s)
I20260620 07:02:45.863237 21888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:45.866986 21955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:45.871217 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21888
I20260620 07:02:45.871414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:45.871961 388 external_mini_cluster.cc:1428] Setting key ab094570bf0a1aef3f97ec03d7fe961b
I20260620 07:02:45.878237 21955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d: Bootstrap starting.
I20260620 07:02:45.881844 21955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:45.883070 21955 log.cc:824] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:45.886245 21955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d: No bootstrap required, opened a new log
I20260620 07:02:45.886292 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:45.873779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:52597 (local address 127.0.0.1:39845)
0620 07:02:45.874499 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:45.874510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:45.874582 (+ 72us) server_negotiation.cc:407] Connection header received
0620 07:02:45.877019 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:45.877044 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:45.877464 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:45.877792 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:45.878423 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:45.879803 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:45.881653 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:45.882440 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:45.882629 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:45.883908 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:45.883919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:45.883938 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:45.884682 (+ 744us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:45.884758 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:45.884784 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:45.884903 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:45.885197 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 07:02:45.885327 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":300,"threads_started":1}
I20260620 07:02:45.889973 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:45.894032 21955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 07:02:45.894477 21955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:45.894608 21955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aafd2ef1a3a40ab9af9f1b35151c33d, State: Initialized, Role: FOLLOWER
I20260620 07:02:45.895298 21955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 07:02:45.895541 21955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:45.895699 21955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:45.895876 21955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:45.898387 21955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 07:02:45.899044 21955 leader_election.cc:302] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aafd2ef1a3a40ab9af9f1b35151c33d; no voters:
I20260620 07:02:45.899724 21955 leader_election.cc:288] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:45.899853 21965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:45.900499 21965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [term 1 LEADER]: Becoming Leader. State: Replica: 9aafd2ef1a3a40ab9af9f1b35151c33d, State: Running, Role: LEADER
I20260620 07:02:45.901306 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:45.894117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:35717 (local address 127.0.0.1:39845)
0620 07:02:45.894294 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:45.894301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:45.894710 (+ 409us) server_negotiation.cc:407] Connection header received
0620 07:02:45.894918 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:45.894923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:45.895010 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:45.895118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:45.895559 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:45.896647 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:45.897950 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:45.898594 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:45.898703 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:45.900181 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:45.900188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:45.900192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:45.900214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:45.900265 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:45.900293 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:45.900420 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:45.900908 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 07:02:45.900985 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:02:45.901348 21965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } }
I20260620 07:02:45.903164 21955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:45.904917 21967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } } }
I20260620 07:02:45.905313 21967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:45.904889 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aafd2ef1a3a40ab9af9f1b35151c33d. Latest consensus state: current_term: 1 leader_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aafd2ef1a3a40ab9af9f1b35151c33d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39845 } } }
I20260620 07:02:45.906261 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:45.908350 21970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:45.913245 21970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:45.922690 21970 catalog_manager.cc:1347] Generated new cluster ID: a42781dc3b2b4c479d05e7f90abf4b3a
I20260620 07:02:45.922963 21970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:45.934908 21970 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:45.936422 21970 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:45.943931 21970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aafd2ef1a3a40ab9af9f1b35151c33d: Generated new TSK 0
I20260620 07:02:45.944736 21970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:45.951730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21888
W20260620 07:02:45.953495 21950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:52597
W20260620 07:02:45.954344 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:45.954521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21888
W20260620 07:02:45.968081 21873 connection.cc:535] client connection to 127.0.0.1:39845 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39845): Connection reset by peer (error 104)
2026-06-20T07:02:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:46 dist-test-slave-mz20 krb5kdc[21986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:46 dist-test-slave-mz20 krb5kdc[21986](info): set up 2 sockets
Jun 20 07:02:46 dist-test-slave-mz20 krb5kdc[21986](info): commencing operation
krb5kdc: starting...
W20260620 07:02:46.990751 388 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:47 dist-test-slave-mz20 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 07:02:47.229380 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:47.550132 21999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:47.550447 21999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:47.550606 21999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:47.557937 21999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:47.558085 21999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:47.558166 21999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:47.558259 21999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:47.558321 21999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:47.568352 21999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36793
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:47.569628 21999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:47.570994 21999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:47.582623 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:47.582753 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:47.582593 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:47.583370 21999 server_base.cc:1034] running on GCE node
I20260620 07:02:47.584143 21999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:47.585593 21999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:47.586828 21999 hybrid_clock.cc:648] HybridClock initialized: now 1781938967586765 us; error 53 us; skew 500 ppm
I20260620 07:02:47.587189 21999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:47 dist-test-slave-mz20 krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:47.590369 21999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 07:02:47.591938 21999 webserver.cc:460] Webserver started at http://127.0.0.1:37363/ using document root <none> and password file <none>
I20260620 07:02:47.592608 21999 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:47.592707 21999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:47.593009 21999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:47.596084 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "c0ae6cb3673640d583b9c86f7ea4726d"
format_stamp: "Formatted at 2026-06-20 07:02:47 on dist-test-slave-mz20"
server_key: "5e2d0b093e9afd91078ae5525cd86eb6"
server_key_iv: "441186094d9e28b0185d4f160fd15694"
server_key_version: "encryptionkey@0"
I20260620 07:02:47.596859 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0ae6cb3673640d583b9c86f7ea4726d"
format_stamp: "Formatted at 2026-06-20 07:02:47 on dist-test-slave-mz20"
server_key: "5e2d0b093e9afd91078ae5525cd86eb6"
server_key_iv: "441186094d9e28b0185d4f160fd15694"
server_key_version: "encryptionkey@0"
I20260620 07:02:47.602159 21999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 07:02:47.605728 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:47.607197 21999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:47.607364 21999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "c0ae6cb3673640d583b9c86f7ea4726d"
format_stamp: "Formatted at 2026-06-20 07:02:47 on dist-test-slave-mz20"
server_key: "5e2d0b093e9afd91078ae5525cd86eb6"
server_key_iv: "441186094d9e28b0185d4f160fd15694"
server_key_version: "encryptionkey@0"
I20260620 07:02:47.607556 21999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:47.613090 21999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:47.617003 21999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:47.617259 21999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:47.634073 21999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36793
I20260620 07:02:47.634107 22065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36793 every 8 connection(s)
I20260620 07:02:47.635496 21999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:47.639117 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:47.641894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 21999
I20260620 07:02:47.642040 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:47.642436 388 external_mini_cluster.cc:1428] Setting key 7407212314b0d7bb2da0cf7876f2449c
I20260620 07:02:47.649714 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:47.644138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:47587 (local address 127.0.0.1:36793)
0620 07:02:47.644869 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:47.644880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:47.644918 (+ 38us) server_negotiation.cc:407] Connection header received
0620 07:02:47.646511 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:47.646531 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:47.646912 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:47.647241 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:47.648231 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:47.648240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:47.648253 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:47.648274 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:47.648336 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:47.648357 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:47.648528 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:47.648632 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 07:02:47.648814 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":313,"threads_started":1}
I20260620 07:02:47.650651 22066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d: Bootstrap starting.
I20260620 07:02:47.653276 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:47.653800 22066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:47.654780 22066 log.cc:824] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:47.657958 22066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d: No bootstrap required, opened a new log
I20260620 07:02:47.660020 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:47.657556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.24:49217 (local address 127.0.0.1:36793)
0620 07:02:47.657709 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:47.657715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:47.658130 (+ 415us) server_negotiation.cc:407] Connection header received
0620 07:02:47.658313 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:47.658321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:47.658373 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:47.658455 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:47.659398 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:47.659404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:47.659409 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:47.659430 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:47.659472 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:47.659487 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:47.659601 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:47.659693 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 07:02:47.659758 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 07:02:47.665458 22066 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } }
I20260620 07:02:47.665832 22066 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:47.665927 22066 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ae6cb3673640d583b9c86f7ea4726d, State: Initialized, Role: FOLLOWER
I20260620 07:02:47.666517 22066 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } }
I20260620 07:02:47.666808 22066 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:47.666956 22066 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:47.667125 22066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:47.669301 22066 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } }
I20260620 07:02:47.669765 22066 leader_election.cc:302] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0ae6cb3673640d583b9c86f7ea4726d; no voters:
I20260620 07:02:47.670306 22066 leader_election.cc:288] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:47.670426 22076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:47.671131 22076 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [term 1 LEADER]: Becoming Leader. State: Replica: c0ae6cb3673640d583b9c86f7ea4726d, State: Running, Role: LEADER
I20260620 07:02:47.671705 22076 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } }
I20260620 07:02:47.672304 22066 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:47.674311 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0ae6cb3673640d583b9c86f7ea4726d. Latest consensus state: current_term: 1 leader_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } } }
I20260620 07:02:47.674672 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:47.674722 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0ae6cb3673640d583b9c86f7ea4726d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36793 } } }
I20260620 07:02:47.674991 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:47.677174 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:47.681443 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:47.689814 22083 catalog_manager.cc:1347] Generated new cluster ID: ed518cef0423499d924abba4a6f0ddd2
I20260620 07:02:47.690222 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:47.723618 22083 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:47.724033 22083 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:47.734819 22083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0ae6cb3673640d583b9c86f7ea4726d: Generated new TSK 0
I20260620 07:02:47.735698 22083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:47.747184 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21999
W20260620 07:02:47.748265 22061 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.24:47587
W20260620 07:02:47.748977 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 21999: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:47.749109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 21999
2026-06-20T07:02:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1790 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:47 dist-test-slave-mz20 krb5kdc[22097](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:47 dist-test-slave-mz20 krb5kdc[22097](info): set up 2 sockets
Jun 20 07:02:47 dist-test-slave-mz20 krb5kdc[22097](info): commencing operation
krb5kdc: starting...
W20260620 07:02:48.770756 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:48 dist-test-slave-mz20 krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:49.006948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35169
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:49.323426 22110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:49.323755 22110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:49.323897 22110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:49.331238 22110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:49.331374 22110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:49.331451 22110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:49.331543 22110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:49.331601 22110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:49.341568 22110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:49.342937 22110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:49.344223 22110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:49.355561 22115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:49.355505 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:49.355538 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:49.356547 22110 server_base.cc:1034] running on GCE node
I20260620 07:02:49.357388 22110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:49.358892 22110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:49.360108 22110 hybrid_clock.cc:648] HybridClock initialized: now 1781938969360054 us; error 50 us; skew 500 ppm
I20260620 07:02:49.360419 22110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:49 dist-test-slave-mz20 krb5kdc[22097](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:49.366582 22110 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:49.368106 22110 webserver.cc:460] Webserver started at http://127.0.97.62:44257/ using document root <none> and password file <none>
I20260620 07:02:49.368769 22110 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:49.368876 22110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:49.369166 22110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:49.372153 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "e0d3432007194cf29afe2fc080f0481e"
format_stamp: "Formatted at 2026-06-20 07:02:49 on dist-test-slave-mz20"
server_key: "b5094402d97f98187b43b0f18c814a2c"
server_key_iv: "494741657badef5415418b510a91cda6"
server_key_version: "encryptionkey@0"
I20260620 07:02:49.372862 22110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "e0d3432007194cf29afe2fc080f0481e"
format_stamp: "Formatted at 2026-06-20 07:02:49 on dist-test-slave-mz20"
server_key: "b5094402d97f98187b43b0f18c814a2c"
server_key_iv: "494741657badef5415418b510a91cda6"
server_key_version: "encryptionkey@0"
I20260620 07:02:49.377815 22110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:49.381464 22125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:49.382810 22110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:49.382973 22110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "e0d3432007194cf29afe2fc080f0481e"
format_stamp: "Formatted at 2026-06-20 07:02:49 on dist-test-slave-mz20"
server_key: "b5094402d97f98187b43b0f18c814a2c"
server_key_iv: "494741657badef5415418b510a91cda6"
server_key_version: "encryptionkey@0"
I20260620 07:02:49.383162 22110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:49.414023 22110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:49.418920 22110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:49.419242 22110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:49.437059 22110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35169
I20260620 07:02:49.437083 22176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35169 every 8 connection(s)
I20260620 07:02:49.438383 22110 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:49.440292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22110
I20260620 07:02:49.440491 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:49.441063 388 external_mini_cluster.cc:1428] Setting key 9f236e28f355b23251699adba6ab6006
I20260620 07:02:49.442835 22177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 07:02:49 dist-test-slave-mz20 krb5kdc[22097](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:49.456221 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e: Bootstrap starting.
I20260620 07:02:49.459612 22177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:49.460846 22177 log.cc:824] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:49.463991 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:49.442938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35075 (local address 127.0.97.62:35169)
0620 07:02:49.443708 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:49.443719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:49.443758 (+ 39us) server_negotiation.cc:407] Connection header received
0620 07:02:49.445339 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:49.445360 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:49.445741 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:49.446064 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:49.447947 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:49.449289 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:49.451082 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:49.451748 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:49.451878 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:49.456661 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:49.456670 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:49.456684 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:49.456721 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:49.459690 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:49.460623 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:49.460630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:49.460636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:49.460806 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:49.461259 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:49.461263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:49.461265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:49.461973 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:49.462283 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:49.462753 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 07:02:49.463043 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":315,"threads_started":1}
I20260620 07:02:49.464128 22177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e: No bootstrap required, opened a new log
I20260620 07:02:49.470214 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:49.473807 22177 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } }
I20260620 07:02:49.474305 22177 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:49.474418 22177 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d3432007194cf29afe2fc080f0481e, State: Initialized, Role: FOLLOWER
I20260620 07:02:49.475230 22177 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } }
I20260620 07:02:49.475524 22177 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:49.475667 22177 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:49.475857 22177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:49.478430 22177 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } }
I20260620 07:02:49.478976 22177 leader_election.cc:302] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0d3432007194cf29afe2fc080f0481e; no voters:
I20260620 07:02:49.479650 22177 leader_election.cc:288] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:49.479784 22187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:49.481921 22177 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:49.486683 22187 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [term 1 LEADER]: Becoming Leader. State: Replica: e0d3432007194cf29afe2fc080f0481e, State: Running, Role: LEADER
I20260620 07:02:49.487493 22187 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } }
I20260620 07:02:49.490621 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0d3432007194cf29afe2fc080f0481e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } } }
I20260620 07:02:49.491017 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:49.491513 22189 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0d3432007194cf29afe2fc080f0481e. Latest consensus state: current_term: 1 leader_uuid: "e0d3432007194cf29afe2fc080f0481e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0d3432007194cf29afe2fc080f0481e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35169 } } }
I20260620 07:02:49.491772 22189 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:49.492225 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:49.476897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43159 (local address 127.0.97.62:35169)
0620 07:02:49.477072 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:49.477077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:49.477459 (+ 382us) server_negotiation.cc:407] Connection header received
0620 07:02:49.477633 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:49.477639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:49.477700 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:49.477783 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:49.479297 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:49.480437 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:49.481710 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:49.482225 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:49.482331 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:49.484714 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:49.484720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:49.484723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:49.484752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:49.488175 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:49.489005 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:49.489011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:49.489015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:49.489154 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:49.489709 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:49.489718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:49.489721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:49.490231 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:49.491425 (+ 1194us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:49.491613 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 07:02:49.491820 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 07:02:49.498999 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:49.504383 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:49.513923 22196 catalog_manager.cc:1347] Generated new cluster ID: ec8721a681f04f5da6bb4670a2090da9
I20260620 07:02:49.514066 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:49.531205 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:49.533030 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:49.546484 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0d3432007194cf29afe2fc080f0481e: Generated new TSK 0
I20260620 07:02:49.547626 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:49.564543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22110
I20260620 07:02:49.566377 22172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:49.659329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22110
W20260620 07:02:49.673012 22095 connection.cc:535] client connection to 127.0.97.62:35169 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35169): Connection reset by peer (error 104)
2026-06-20T07:02:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:49 dist-test-slave-mz20 krb5kdc[22210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:49 dist-test-slave-mz20 krb5kdc[22210](info): set up 2 sockets
Jun 20 07:02:49 dist-test-slave-mz20 krb5kdc[22210](info): commencing operation
krb5kdc: starting...
W20260620 07:02:50.674713 388 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:50 dist-test-slave-mz20 krb5kdc[22210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:50.915963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44849
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:51.232525 22223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:51.232823 22223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:51.232918 22223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:51.240078 22223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:51.240187 22223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:51.240231 22223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:51.240289 22223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:51.240332 22223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:51.249944 22223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:51.251194 22223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:51.252395 22223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:51.263336 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:51.263496 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:51.263906 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:51.264390 22223 server_base.cc:1034] running on GCE node
I20260620 07:02:51.265147 22223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:51.266530 22223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:51.267781 22223 hybrid_clock.cc:648] HybridClock initialized: now 1781938971267732 us; error 39 us; skew 500 ppm
I20260620 07:02:51.268121 22223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:51 dist-test-slave-mz20 krb5kdc[22210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:51.271229 22223 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:51.272769 22223 webserver.cc:460] Webserver started at http://127.0.97.62:39895/ using document root <none> and password file <none>
I20260620 07:02:51.273427 22223 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:51.273538 22223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:51.273828 22223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:51.276796 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "89f4d8922b694963b6b94b593163b759"
format_stamp: "Formatted at 2026-06-20 07:02:51 on dist-test-slave-mz20"
server_key: "b7f60ccb2531224e2d9ca1ffd96b7239"
server_key_iv: "d263c9c5f08a02abcb66b0d3cb806fb0"
server_key_version: "encryptionkey@0"
I20260620 07:02:51.277583 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "89f4d8922b694963b6b94b593163b759"
format_stamp: "Formatted at 2026-06-20 07:02:51 on dist-test-slave-mz20"
server_key: "b7f60ccb2531224e2d9ca1ffd96b7239"
server_key_iv: "d263c9c5f08a02abcb66b0d3cb806fb0"
server_key_version: "encryptionkey@0"
I20260620 07:02:51.282629 22223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:02:51.286124 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:51.287575 22223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 07:02:51.287746 22223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "89f4d8922b694963b6b94b593163b759"
format_stamp: "Formatted at 2026-06-20 07:02:51 on dist-test-slave-mz20"
server_key: "b7f60ccb2531224e2d9ca1ffd96b7239"
server_key_iv: "d263c9c5f08a02abcb66b0d3cb806fb0"
server_key_version: "encryptionkey@0"
I20260620 07:02:51.287923 22223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:51.302423 22223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:51.306327 22223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:51.306602 22223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:51.323999 22223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44849
I20260620 07:02:51.324046 22289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44849 every 8 connection(s)
I20260620 07:02:51.325378 22223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:51.328737 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22223
I20260620 07:02:51.328938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:51.329474 388 external_mini_cluster.cc:1428] Setting key 9ddc26e10f1b086407b68bd5f3415813
I20260620 07:02:51.329520 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:51.342195 22290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759: Bootstrap starting.
I20260620 07:02:51.342609 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:51.331351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56185 (local address 127.0.97.62:44849)
0620 07:02:51.332105 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:51.332123 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:51.332169 (+ 46us) server_negotiation.cc:407] Connection header received
0620 07:02:51.334240 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:51.334264 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:51.334761 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:51.335147 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:51.335714 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:51.336999 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:51.338296 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:51.339028 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:51.339212 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:51.340392 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:51.340399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:51.340413 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:51.340948 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:51.341026 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:51.341056 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:51.341190 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:51.341456 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 07:02:51.341617 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
I20260620 07:02:51.345778 22290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:51.346571 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:51.346940 22290 log.cc:824] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:51.349848 22290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759: No bootstrap required, opened a new log
I20260620 07:02:51.357585 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:51.350669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46003 (local address 127.0.97.62:44849)
0620 07:02:51.350866 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:51.350872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:51.351342 (+ 470us) server_negotiation.cc:407] Connection header received
0620 07:02:51.351492 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:51.351501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:51.351582 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:51.351706 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:51.352201 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:51.353370 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:51.354733 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:51.355431 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:51.355582 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:51.356731 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:51.356737 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:51.356742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:51.356774 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:51.356820 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:51.356839 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:51.356963 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:51.357176 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 07:02:51.357235 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 07:02:51.359216 22290 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } }
I20260620 07:02:51.359594 22290 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:51.359704 22290 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f4d8922b694963b6b94b593163b759, State: Initialized, Role: FOLLOWER
I20260620 07:02:51.360265 22290 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } }
I20260620 07:02:51.360484 22290 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:51.360605 22290 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:51.360744 22290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:51.363062 22290 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } }
I20260620 07:02:51.363577 22290 leader_election.cc:302] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f4d8922b694963b6b94b593163b759; no voters:
I20260620 07:02:51.364197 22290 leader_election.cc:288] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:51.364351 22300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:51.365439 22300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [term 1 LEADER]: Becoming Leader. State: Replica: 89f4d8922b694963b6b94b593163b759, State: Running, Role: LEADER
I20260620 07:02:51.365980 22300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } }
I20260620 07:02:51.366492 22290 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:51.368615 22301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f4d8922b694963b6b94b593163b759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } } }
I20260620 07:02:51.369021 22301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:51.371413 22302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f4d8922b694963b6b94b593163b759. Latest consensus state: current_term: 1 leader_uuid: "89f4d8922b694963b6b94b593163b759" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f4d8922b694963b6b94b593163b759" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44849 } } }
I20260620 07:02:51.371734 22302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:51.372095 22309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:51.376180 22309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:51.384666 22309 catalog_manager.cc:1347] Generated new cluster ID: fbd1c4037f0e4ec688819a3233c6339f
I20260620 07:02:51.384785 22309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:51.417629 22309 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:51.419082 22309 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:51.426368 22309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f4d8922b694963b6b94b593163b759: Generated new TSK 0
I20260620 07:02:51.427182 22309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:51.433511 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22223
W20260620 07:02:51.434641 22285 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56185
W20260620 07:02:51.435444 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22223: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:51.435580 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22223
W20260620 07:02:51.449309 22208 connection.cc:535] client connection to 127.0.97.62:44849 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44849): Connection reset by peer (error 104)
2026-06-20T07:02:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1776 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:51 dist-test-slave-mz20 krb5kdc[22321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:51 dist-test-slave-mz20 krb5kdc[22321](info): set up 2 sockets
Jun 20 07:02:51 dist-test-slave-mz20 krb5kdc[22321](info): commencing operation
krb5kdc: starting...
W20260620 07:02:52.442739 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:52 dist-test-slave-mz20 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:52.687599 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37073
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:53.015631 22334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:53.015980 22334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:53.016136 22334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:53.023589 22334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:53.023722 22334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:53.023811 22334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:53.023893 22334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:53.023952 22334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:53.033994 22334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:53.035284 22334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:53.036545 22334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:53.048523 22340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:53.048444 22342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:53.048559 22339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:53.049250 22334 server_base.cc:1034] running on GCE node
I20260620 07:02:53.050084 22334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:53.051704 22334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:53.052939 22334 hybrid_clock.cc:648] HybridClock initialized: now 1781938973052883 us; error 50 us; skew 500 ppm
I20260620 07:02:53.053390 22334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:53 dist-test-slave-mz20 krb5kdc[22321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:53.056571 22334 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:53.058121 22334 webserver.cc:460] Webserver started at http://127.0.97.62:36093/ using document root <none> and password file <none>
I20260620 07:02:53.058857 22334 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:53.058979 22334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:53.059274 22334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:53.062281 22334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "f02f19fead8e4ac98e6f6cb249890201"
format_stamp: "Formatted at 2026-06-20 07:02:53 on dist-test-slave-mz20"
server_key: "a6190570b683268b0dab528975c13d72"
server_key_iv: "d590dabe2736bd4e1be7fbac30cee37a"
server_key_version: "encryptionkey@0"
I20260620 07:02:53.063109 22334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "f02f19fead8e4ac98e6f6cb249890201"
format_stamp: "Formatted at 2026-06-20 07:02:53 on dist-test-slave-mz20"
server_key: "a6190570b683268b0dab528975c13d72"
server_key_iv: "d590dabe2736bd4e1be7fbac30cee37a"
server_key_version: "encryptionkey@0"
I20260620 07:02:53.068895 22334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 07:02:53.072732 22349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:53.074244 22334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:53.074458 22334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "f02f19fead8e4ac98e6f6cb249890201"
format_stamp: "Formatted at 2026-06-20 07:02:53 on dist-test-slave-mz20"
server_key: "a6190570b683268b0dab528975c13d72"
server_key_iv: "d590dabe2736bd4e1be7fbac30cee37a"
server_key_version: "encryptionkey@0"
I20260620 07:02:53.074714 22334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:53.080631 22334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:53.084733 22334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:53.085026 22334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:53.102382 22334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37073
I20260620 07:02:53.102424 22400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37073 every 8 connection(s)
I20260620 07:02:53.103729 22334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:53.107278 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:53.111608 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22334
I20260620 07:02:53.111794 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:53.112290 388 external_mini_cluster.cc:1428] Setting key 8c332f5a9ca90ca1278178a35feb1758
I20260620 07:02:53.119586 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201: Bootstrap starting.
I20260620 07:02:53.121529 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:53.113976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41099 (local address 127.0.97.62:37073)
0620 07:02:53.114780 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:53.114795 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:53.114846 (+ 51us) server_negotiation.cc:407] Connection header received
0620 07:02:53.117728 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:53.117768 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:53.118223 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:53.118670 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:53.119623 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:53.119630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:53.119647 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:53.119664 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:53.119736 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:53.119779 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:53.119994 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:53.120124 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 07:02:53.120315 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":315,"threads_started":1}
I20260620 07:02:53.123387 22401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:53.124554 22401 log.cc:824] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:53.125303 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:53.127442 22401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201: No bootstrap required, opened a new log
I20260620 07:02:53.131897 22404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:53.129461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41677 (local address 127.0.97.62:37073)
0620 07:02:53.129602 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:53.129606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:53.130057 (+ 451us) server_negotiation.cc:407] Connection header received
0620 07:02:53.130217 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:53.130221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:53.130272 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:53.130360 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:53.131259 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:53.131264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:53.131270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:53.131288 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:53.131325 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:53.131340 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:53.131441 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:53.131537 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 07:02:53.131602 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 07:02:53.136032 22401 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } }
I20260620 07:02:53.136425 22401 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:53.136518 22401 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02f19fead8e4ac98e6f6cb249890201, State: Initialized, Role: FOLLOWER
I20260620 07:02:53.137069 22401 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } }
I20260620 07:02:53.137297 22401 raft_consensus.cc:388] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:53.137428 22401 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:53.137583 22401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:53.139870 22401 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } }
I20260620 07:02:53.140463 22401 leader_election.cc:302] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f02f19fead8e4ac98e6f6cb249890201; no voters:
I20260620 07:02:53.141140 22401 leader_election.cc:288] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:53.141295 22411 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:53.141923 22411 raft_consensus.cc:686] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [term 1 LEADER]: Becoming Leader. State: Replica: f02f19fead8e4ac98e6f6cb249890201, State: Running, Role: LEADER
I20260620 07:02:53.142460 22411 consensus_queue.cc:237] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } }
I20260620 07:02:53.143153 22401 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:53.144990 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f02f19fead8e4ac98e6f6cb249890201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } } }
I20260620 07:02:53.145385 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:53.146066 22413 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02f19fead8e4ac98e6f6cb249890201. Latest consensus state: current_term: 1 leader_uuid: "f02f19fead8e4ac98e6f6cb249890201" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02f19fead8e4ac98e6f6cb249890201" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37073 } } }
I20260620 07:02:53.146340 22413 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:53.148002 22416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:53.153430 22416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:53.162526 22416 catalog_manager.cc:1347] Generated new cluster ID: 177efff868614041936585bf01aed781
I20260620 07:02:53.162683 22416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:53.181305 22416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:53.181830 22416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:53.190428 22416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f02f19fead8e4ac98e6f6cb249890201: Generated new TSK 0
I20260620 07:02:53.191551 22416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:53.206429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22334
W20260620 07:02:53.207512 22396 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41099
W20260620 07:02:53.208268 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22334: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:53.208389 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22334
2026-06-20T07:02:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1772 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:53 dist-test-slave-mz20 krb5kdc[22432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:53 dist-test-slave-mz20 krb5kdc[22432](info): set up 2 sockets
Jun 20 07:02:53 dist-test-slave-mz20 krb5kdc[22432](info): commencing operation
krb5kdc: starting...
W20260620 07:02:54.235176 388 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:54 dist-test-slave-mz20 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:54.540433 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45353
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:54.920969 22445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:54.921341 22445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:54.921443 22445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:54.929301 22445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:54.929440 22445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:54.929491 22445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:54.929557 22445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:54.929596 22445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:54.940387 22445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:54.941939 22445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:54.943459 22445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:54.956247 22450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:54.956184 22453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:54.956233 22451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:54.957278 22445 server_base.cc:1034] running on GCE node
I20260620 07:02:54.958103 22445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:54.959757 22445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:54.960995 22445 hybrid_clock.cc:648] HybridClock initialized: now 1781938974960947 us; error 37 us; skew 500 ppm
I20260620 07:02:54.961426 22445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:54 dist-test-slave-mz20 krb5kdc[22432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:54.964890 22445 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:54.966612 22445 webserver.cc:460] Webserver started at http://127.0.97.62:43665/ using document root <none> and password file <none>
I20260620 07:02:54.967392 22445 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:54.967504 22445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:54.967831 22445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:54.971269 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "61533fafb7ff439586e4fe5d97ba60b3"
format_stamp: "Formatted at 2026-06-20 07:02:54 on dist-test-slave-mz20"
server_key: "d4efffce07e1b5d0651735663bf8379b"
server_key_iv: "9ac78103c988602d45b48b7f19fe1850"
server_key_version: "encryptionkey@0"
I20260620 07:02:54.972126 22445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "61533fafb7ff439586e4fe5d97ba60b3"
format_stamp: "Formatted at 2026-06-20 07:02:54 on dist-test-slave-mz20"
server_key: "d4efffce07e1b5d0651735663bf8379b"
server_key_iv: "9ac78103c988602d45b48b7f19fe1850"
server_key_version: "encryptionkey@0"
I20260620 07:02:54.977857 22445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:54.981693 22460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:54.983198 22445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 07:02:54.983371 22445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "61533fafb7ff439586e4fe5d97ba60b3"
format_stamp: "Formatted at 2026-06-20 07:02:54 on dist-test-slave-mz20"
server_key: "d4efffce07e1b5d0651735663bf8379b"
server_key_iv: "9ac78103c988602d45b48b7f19fe1850"
server_key_version: "encryptionkey@0"
I20260620 07:02:54.983558 22445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:55.017995 22445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:55.022809 22445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:55.023111 22445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:55.042573 22445 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45353
I20260620 07:02:55.042626 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45353 every 8 connection(s)
I20260620 07:02:55.044173 22445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:55.048578 22512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:55.050884 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22445
I20260620 07:02:55.051101 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:55.051708 388 external_mini_cluster.cc:1428] Setting key fec5d5e42dcb9ffa4f3d1f4c11d21db1
I20260620 07:02:55.062748 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3: Bootstrap starting.
I20260620 07:02:55.066802 22512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:55.067938 22512 log.cc:824] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3: Log is configured to *not* fsync() on all Append() calls
Jun 20 07:02:55 dist-test-slave-mz20 krb5kdc[22432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:02:55.071513 22512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3: No bootstrap required, opened a new log
I20260620 07:02:55.077606 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:55.053997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47315 (local address 127.0.97.62:45353)
0620 07:02:55.055033 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:55.055052 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:55.055444 (+ 392us) server_negotiation.cc:407] Connection header received
0620 07:02:55.057924 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:55.057955 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:55.058518 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:55.059026 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:55.060413 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:55.062389 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:55.063871 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:55.064955 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:55.065160 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:55.069737 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:55.069746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:55.069763 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:55.069804 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:55.073083 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:55.073907 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:55.073913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:55.073923 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:55.074117 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:55.074607 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:55.074611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:55.074613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:55.075250 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:55.075523 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:55.075935 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 07:02:55.076354 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":426,"threads_started":1}
I20260620 07:02:55.080724 22512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260620 07:02:55.081148 22512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:55.081262 22512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61533fafb7ff439586e4fe5d97ba60b3, State: Initialized, Role: FOLLOWER
I20260620 07:02:55.081929 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:55.081964 22512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260620 07:02:55.082260 22512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:55.082384 22512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:55.082553 22512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:55.085906 22512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260620 07:02:55.086668 22512 leader_election.cc:302] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61533fafb7ff439586e4fe5d97ba60b3; no voters:
I20260620 07:02:55.087436 22512 leader_election.cc:288] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:55.087625 22521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:55.088927 22521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [term 1 LEADER]: Becoming Leader. State: Replica: 61533fafb7ff439586e4fe5d97ba60b3, State: Running, Role: LEADER
I20260620 07:02:55.089651 22521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } }
I20260620 07:02:55.090270 22512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:55.092738 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61533fafb7ff439586e4fe5d97ba60b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } } }
I20260620 07:02:55.093139 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:55.094391 22524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61533fafb7ff439586e4fe5d97ba60b3. Latest consensus state: current_term: 1 leader_uuid: "61533fafb7ff439586e4fe5d97ba60b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61533fafb7ff439586e4fe5d97ba60b3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45353 } } }
I20260620 07:02:55.095266 22524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:55.108083 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:55.087774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50591 (local address 127.0.97.62:45353)
0620 07:02:55.087996 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:55.088004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:55.089814 (+ 1810us) server_negotiation.cc:407] Connection header received
0620 07:02:55.090025 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:55.090031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:55.090122 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:55.090229 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:55.092220 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:55.093578 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:55.095611 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:55.096381 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:55.096488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:55.100111 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:55.100119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:55.100124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:02:55.100170 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:55.104226 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:55.105185 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:55.105192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:55.105196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:55.105359 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:02:55.105935 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:02:55.105941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:02:55.105945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:02:55.106379 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:55.106678 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:55.107310 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 07:02:55.107597 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
I20260620 07:02:55.110846 22527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:55.117265 22527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:55.128824 22527 catalog_manager.cc:1347] Generated new cluster ID: ab151d39d17e4060821e8228c8e2ec97
I20260620 07:02:55.128961 22527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:55.145049 22527 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:55.146633 22527 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:55.157159 22527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61533fafb7ff439586e4fe5d97ba60b3: Generated new TSK 0
I20260620 07:02:55.158244 22527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:55.165347 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22445
I20260620 07:02:55.166397 22507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:02:55.264966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22445
2026-06-20T07:02:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:55 dist-test-slave-mz20 krb5kdc[22545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:55 dist-test-slave-mz20 krb5kdc[22545](info): set up 2 sockets
Jun 20 07:02:55 dist-test-slave-mz20 krb5kdc[22545](info): commencing operation
krb5kdc: starting...
W20260620 07:02:56.310995 388 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:56 dist-test-slave-mz20 krb5kdc[22545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:56.554193 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32917
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:56.904800 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:56.905113 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:56.905247 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:56.912695 22558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:56.912822 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:56.912876 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:56.912961 22558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:56.913023 22558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:56.923197 22558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:56.924540 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:56.925846 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:56.937325 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:56.937299 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:56.937706 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:56.938143 22558 server_base.cc:1034] running on GCE node
I20260620 07:02:56.939023 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:56.940462 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:56.941684 22558 hybrid_clock.cc:648] HybridClock initialized: now 1781938976941635 us; error 36 us; skew 500 ppm
I20260620 07:02:56.942024 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:56 dist-test-slave-mz20 krb5kdc[22545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:56.945235 22558 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:56.946870 22558 webserver.cc:460] Webserver started at http://127.0.97.62:46247/ using document root <none> and password file <none>
I20260620 07:02:56.947624 22558 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:56.947739 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:56.948022 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:56.951144 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "319f86a2b15945969a33281dad6103d2"
format_stamp: "Formatted at 2026-06-20 07:02:56 on dist-test-slave-mz20"
server_key: "942f02e3ebd1b0ed8ca890e4d0f863bd"
server_key_iv: "19e5475a9caf43963bc4ffa2b34ad616"
server_key_version: "encryptionkey@0"
I20260620 07:02:56.951926 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "319f86a2b15945969a33281dad6103d2"
format_stamp: "Formatted at 2026-06-20 07:02:56 on dist-test-slave-mz20"
server_key: "942f02e3ebd1b0ed8ca890e4d0f863bd"
server_key_iv: "19e5475a9caf43963bc4ffa2b34ad616"
server_key_version: "encryptionkey@0"
I20260620 07:02:56.957600 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:56.961155 22573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:56.962436 22558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 07:02:56.962641 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "319f86a2b15945969a33281dad6103d2"
format_stamp: "Formatted at 2026-06-20 07:02:56 on dist-test-slave-mz20"
server_key: "942f02e3ebd1b0ed8ca890e4d0f863bd"
server_key_iv: "19e5475a9caf43963bc4ffa2b34ad616"
server_key_version: "encryptionkey@0"
I20260620 07:02:56.962836 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:56.991962 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:56.996266 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:56.996506 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:57.013382 22558 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32917
I20260620 07:02:57.013411 22624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32917 every 8 connection(s)
I20260620 07:02:57.014711 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:57.018496 22625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:57.018824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22558
I20260620 07:02:57.019008 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:57.019544 388 external_mini_cluster.cc:1428] Setting key be0528c9c1fb9ac7a682bacefad24997
I20260620 07:02:57.034263 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2: Bootstrap starting.
I20260620 07:02:57.034327 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:57.021298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53429 (local address 127.0.97.62:32917)
0620 07:02:57.022060 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:57.022074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:57.022123 (+ 49us) server_negotiation.cc:407] Connection header received
0620 07:02:57.024263 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:57.024294 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:57.024826 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:57.025302 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:57.025820 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:57.027808 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:57.029195 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:57.030184 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:57.030357 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:57.031546 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:57.031556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:57.031575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:57.032321 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:57.032417 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:57.032447 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:57.032560 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:57.032787 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 07:02:57.033070 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":357,"threads_started":1}
I20260620 07:02:57.037958 22625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:57.038447 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:57.039193 22625 log.cc:824] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:57.042865 22625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2: No bootstrap required, opened a new log
I20260620 07:02:57.050506 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:57.043533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42269 (local address 127.0.97.62:32917)
0620 07:02:57.043713 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:57.043719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:57.044268 (+ 549us) server_negotiation.cc:407] Connection header received
0620 07:02:57.044415 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:57.044420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:57.044509 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:57.044624 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:57.045097 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:57.046181 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:57.047564 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:02:57.048282 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:02:57.048416 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:02:57.049678 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:57.049684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:57.049689 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:57.049716 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:57.049764 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:57.049788 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:57.049897 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:57.050058 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 07:02:57.050154 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 07:02:57.052294 22625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 07:02:57.052899 22625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:57.053025 22625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 319f86a2b15945969a33281dad6103d2, State: Initialized, Role: FOLLOWER
I20260620 07:02:57.053668 22625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 07:02:57.053915 22625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:57.054064 22625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:57.054245 22625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:57.056496 22625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 07:02:57.057029 22625 leader_election.cc:302] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 319f86a2b15945969a33281dad6103d2; no voters:
I20260620 07:02:57.057734 22625 leader_election.cc:288] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:57.057945 22635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:57.059108 22635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [term 1 LEADER]: Becoming Leader. State: Replica: 319f86a2b15945969a33281dad6103d2, State: Running, Role: LEADER
I20260620 07:02:57.059639 22635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } }
I20260620 07:02:57.060097 22625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:57.062162 22637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 319f86a2b15945969a33281dad6103d2. Latest consensus state: current_term: 1 leader_uuid: "319f86a2b15945969a33281dad6103d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } } }
I20260620 07:02:57.062577 22637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:57.063298 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "319f86a2b15945969a33281dad6103d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "319f86a2b15945969a33281dad6103d2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32917 } } }
I20260620 07:02:57.063759 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:57.064603 22641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:57.071071 22641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:57.080526 22641 catalog_manager.cc:1347] Generated new cluster ID: 3def2d41a24842ac9a141cc1f1a6df3c
I20260620 07:02:57.080663 22641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:57.111423 22641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:57.113315 22641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:57.128003 22641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 319f86a2b15945969a33281dad6103d2: Generated new TSK 0
I20260620 07:02:57.128857 22641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:57.144877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22558
W20260620 07:02:57.145974 22620 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53429
W20260620 07:02:57.146699 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22558: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:57.146833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22558
2026-06-20T07:02:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:57 dist-test-slave-mz20 krb5kdc[22656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:57 dist-test-slave-mz20 krb5kdc[22656](info): set up 2 sockets
Jun 20 07:02:57 dist-test-slave-mz20 krb5kdc[22656](info): commencing operation
krb5kdc: starting...
W20260620 07:02:58.162897 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:02:58 dist-test-slave-mz20 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:02:58.413564 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33391
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:02:58.738922 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:02:58.739219 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:02:58.739327 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:02:58.746671 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:02:58.746784 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:02:58.746855 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:02:58.746938 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:02:58.746997 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:02:58.756808 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:02:58.758075 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:02:58.759356 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:02:58.770749 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:58.770830 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:02:58.770804 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:02:58.771557 22669 server_base.cc:1034] running on GCE node
I20260620 07:02:58.772359 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:02:58.773811 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:02:58.775039 22669 hybrid_clock.cc:648] HybridClock initialized: now 1781938978774972 us; error 53 us; skew 500 ppm
I20260620 07:02:58.775410 22669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:02:58 dist-test-slave-mz20 krb5kdc[22656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:02:58.778504 22669 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:02:58.780027 22669 webserver.cc:460] Webserver started at http://127.0.97.62:36417/ using document root <none> and password file <none>
I20260620 07:02:58.780699 22669 fs_manager.cc:359] Metadata directory not provided
I20260620 07:02:58.780813 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:02:58.781091 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:02:58.784078 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "cacef8784fe147dfba2bae1ab8fa2107"
format_stamp: "Formatted at 2026-06-20 07:02:58 on dist-test-slave-mz20"
server_key: "afcc5c1cc4ae6545a54c807f0c74d756"
server_key_iv: "62abfe50be272e151281fc66a38d7fb4"
server_key_version: "encryptionkey@0"
I20260620 07:02:58.784844 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "cacef8784fe147dfba2bae1ab8fa2107"
format_stamp: "Formatted at 2026-06-20 07:02:58 on dist-test-slave-mz20"
server_key: "afcc5c1cc4ae6545a54c807f0c74d756"
server_key_iv: "62abfe50be272e151281fc66a38d7fb4"
server_key_version: "encryptionkey@0"
I20260620 07:02:58.789937 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 07:02:58.793318 22684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:02:58.794700 22669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:02:58.794863 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "cacef8784fe147dfba2bae1ab8fa2107"
format_stamp: "Formatted at 2026-06-20 07:02:58 on dist-test-slave-mz20"
server_key: "afcc5c1cc4ae6545a54c807f0c74d756"
server_key_iv: "62abfe50be272e151281fc66a38d7fb4"
server_key_version: "encryptionkey@0"
I20260620 07:02:58.795044 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:02:58.800864 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:02:58.805477 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:02:58.805812 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:02:58.822925 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33391
I20260620 07:02:58.822973 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33391 every 8 connection(s)
I20260620 07:02:58.824297 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:02:58.827677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22669
I20260620 07:02:58.827843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:02:58.828373 388 external_mini_cluster.cc:1428] Setting key 85e67636ee844f6f8f66aa55265efd7c
I20260620 07:02:58.828367 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:02:58.836853 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:58.830407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59579 (local address 127.0.97.62:33391)
0620 07:02:58.831513 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:58.831526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:58.831571 (+ 45us) server_negotiation.cc:407] Connection header received
0620 07:02:58.833263 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:58.833287 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:58.833660 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:58.834027 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:58.835208 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:58.835215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:58.835231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:58.835251 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:58.835314 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:58.835337 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:58.835527 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:58.835640 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 07:02:58.835825 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":567,"threads_started":1}
I20260620 07:02:58.840900 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:02:58.842126 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107: Bootstrap starting.
I20260620 07:02:58.845433 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107: Neither blocks nor log segments found. Creating new log.
I20260620 07:02:58.846494 22736 log.cc:824] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107: Log is configured to *not* fsync() on all Append() calls
I20260620 07:02:58.847568 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:02:58.845110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44629 (local address 127.0.97.62:33391)
0620 07:02:58.845308 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:02:58.845315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:02:58.845703 (+ 388us) server_negotiation.cc:407] Connection header received
0620 07:02:58.845850 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:02:58.845854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:02:58.845906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:02:58.845985 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:02:58.846936 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:02:58.846941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:02:58.846946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 07:02:58.846962 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:02:58.847004 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 07:02:58.847019 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:02:58.847136 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 07:02:58.847249 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 07:02:58.847315 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 07:02:58.849484 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107: No bootstrap required, opened a new log
I20260620 07:02:58.857144 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260620 07:02:58.857583 22736 raft_consensus.cc:374] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 07:02:58.857659 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacef8784fe147dfba2bae1ab8fa2107, State: Initialized, Role: FOLLOWER
I20260620 07:02:58.858233 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260620 07:02:58.858440 22736 raft_consensus.cc:388] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 07:02:58.858561 22736 raft_consensus.cc:482] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 07:02:58.858707 22736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:02:58.860837 22736 raft_consensus.cc:504] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260620 07:02:58.861299 22736 leader_election.cc:302] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cacef8784fe147dfba2bae1ab8fa2107; no voters:
I20260620 07:02:58.861855 22736 leader_election.cc:288] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 07:02:58.861995 22746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:02:58.862707 22746 raft_consensus.cc:686] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [term 1 LEADER]: Becoming Leader. State: Replica: cacef8784fe147dfba2bae1ab8fa2107, State: Running, Role: LEADER
I20260620 07:02:58.863236 22746 consensus_queue.cc:237] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } }
I20260620 07:02:58.863930 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:02:58.866194 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cacef8784fe147dfba2bae1ab8fa2107. Latest consensus state: current_term: 1 leader_uuid: "cacef8784fe147dfba2bae1ab8fa2107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } }
I20260620 07:02:58.866659 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:58.867151 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cacef8784fe147dfba2bae1ab8fa2107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cacef8784fe147dfba2bae1ab8fa2107" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33391 } } }
I20260620 07:02:58.867432 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107 [sys.catalog]: This master's current role is: LEADER
I20260620 07:02:58.870664 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:02:58.876099 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:02:58.884692 22752 catalog_manager.cc:1347] Generated new cluster ID: 2128fb771666449c9deface5ad9ba7a9
I20260620 07:02:58.884814 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:02:58.915237 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:02:58.915692 22752 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:02:58.928694 22752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cacef8784fe147dfba2bae1ab8fa2107: Generated new TSK 0
I20260620 07:02:58.929570 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 07:02:58.938030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22669
W20260620 07:02:58.939080 22731 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59579
W20260620 07:02:58.939750 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22669: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 07:02:58.939869 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22669
2026-06-20T07:02:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1792 ms)
[----------] 18 tests from AuthTokenIssuingTest (33370 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:02:59 dist-test-slave-mz20 krb5kdc[22767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:02:59 dist-test-slave-mz20 krb5kdc[22767](info): set up 2 sockets
Jun 20 07:02:59 dist-test-slave-mz20 krb5kdc[22767](info): commencing operation
krb5kdc: starting...
W20260620 07:02:59.966952 388 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:03:00 dist-test-slave-mz20 krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:03:00.228125 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:00.555004 22780 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:00.555354 22780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:00.555457 22780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:00.555581 22780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:00.562829 22780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:00.562960 22780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:00.563050 22780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:00.563113 22780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:00.563202 22780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:00.573320 22780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:00.574663 22780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:00.575974 22780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:00.587574 22785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:00.587743 22788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:00.587674 22786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:00.588575 22780 server_base.cc:1034] running on GCE node
I20260620 07:03:00.589351 22780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:00.590903 22780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:00.592166 22780 hybrid_clock.cc:648] HybridClock initialized: now 1781938980592108 us; error 34 us; skew 500 ppm
I20260620 07:03:00.592489 22780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:00 dist-test-slave-mz20 krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:00.606029 22780 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:03:00.607703 22780 webserver.cc:460] Webserver started at http://127.0.97.62:40717/ using document root <none> and password file <none>
I20260620 07:03:00.608404 22780 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:00.608497 22780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:00.608758 22780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:00.611923 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "62862451b2b14ead80d3ff87ea5b57de"
format_stamp: "Formatted at 2026-06-20 07:03:00 on dist-test-slave-mz20"
server_key: "5c853c7f4fad0df871747c051e47d9e0"
server_key_iv: "1fcc9b0b9b1e338d54b4e1ec04f6473f"
server_key_version: "encryptionkey@0"
I20260620 07:03:00.612679 22780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "62862451b2b14ead80d3ff87ea5b57de"
format_stamp: "Formatted at 2026-06-20 07:03:00 on dist-test-slave-mz20"
server_key: "5c853c7f4fad0df871747c051e47d9e0"
server_key_iv: "1fcc9b0b9b1e338d54b4e1ec04f6473f"
server_key_version: "encryptionkey@0"
I20260620 07:03:00.617841 22780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 07:03:00.621474 22795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:00.623070 22780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 07:03:00.623230 22780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "62862451b2b14ead80d3ff87ea5b57de"
format_stamp: "Formatted at 2026-06-20 07:03:00 on dist-test-slave-mz20"
server_key: "5c853c7f4fad0df871747c051e47d9e0"
server_key_iv: "1fcc9b0b9b1e338d54b4e1ec04f6473f"
server_key_version: "encryptionkey@0"
I20260620 07:03:00.623452 22780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:00.651002 22780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:00.655435 22780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:00.655710 22780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:00.673724 22780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40209
I20260620 07:03:00.673743 22846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40209 every 8 connection(s)
I20260620 07:03:00.675191 22780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:00.678872 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:00.683023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22780
I20260620 07:03:00.683234 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:03:00.684000 388 external_mini_cluster.cc:1428] Setting key 76af1655658727d25b5e562f346df3ca
I20260620 07:03:00.686625 22847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:00 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 07:03:00.710798 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.690267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40209 (local address 127.0.0.1:34398)
0620 07:03:00.691659 (+ 1392us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.691698 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:00.693958 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:00.694912 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:00.694917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:00.695453 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:00.695773 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:00.695789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:00.697861 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:00.697865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:00.699044 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:00.699055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:00.700160 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:00.700165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:00.700287 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:00.700926 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:00.700951 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:00.703295 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:00.706520 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:00.706572 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:00.706586 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:00.707145 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:00.707648 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:00.707653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:00.707657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:00.707853 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:00.708647 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:00.708656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:00.708915 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 07:03:00.709204 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 07:03:00.709667 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":3,"thread_start_us":934,"threads_started":1}
I20260620 07:03:00.710819 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.690267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.97.62:40209)
0620 07:03:00.691130 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:00.691144 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:00.691837 (+ 693us) server_negotiation.cc:407] Connection header received
0620 07:03:00.694330 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:00.694338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:00.694669 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:00.694893 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:00.696016 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:00.697697 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:00.699177 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:00.699916 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:00.700097 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:00.703529 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:00.703537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:00.703548 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:00.703575 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:00.706359 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:00.707370 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:00.707377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:00.707379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:00.707529 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:00.707961 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:00.707964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:00.707966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:00.708494 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:00.708643 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:00.709363 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 07:03:00.709779 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":641,"thread_start_us":391,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 07:03:00.713531 22847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:00.714619 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.714208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40056)
0620 07:03:00.714378 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.714480 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 07:03:00.714902 22799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
I20260620 07:03:00.716308 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.715981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40062)
0620 07:03:00.716116 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.716191 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:03:00.716516 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33829: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 07:03:00.741094 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33829
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:03:00.765851 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } attempt: 1
I20260620 07:03:00.767076 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.766606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40078)
0620 07:03:00.766833 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.766939 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 07:03:00.768095 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.767719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40094)
0620 07:03:00.767899 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.767969 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:03:00.768287 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33829: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
I20260620 07:03:00.836565 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } attempt: 2
I20260620 07:03:00.837791 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.837277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40096)
0620 07:03:00.837550 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.837642 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 07:03:00.838824 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.838445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40106)
0620 07:03:00.838628 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.838699 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 07:03:00.839025 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33829: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
I20260620 07:03:00.930299 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } attempt: 3
I20260620 07:03:00.931653 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.931068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40114)
0620 07:03:00.931306 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.931493 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 07:03:00.932727 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:00.932355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40118)
0620 07:03:00.932513 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:00.932588 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 07:03:00.932940 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33829: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
W20260620 07:03:01.067354 22854 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:01.067703 22854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:01.067808 22854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:01.067929 22854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:01.075305 22854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:01.075413 22854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:01.075498 22854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:01.075567 22854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:01.075659 22854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:01.076167 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } attempt: 4
I20260620 07:03:01.077301 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.076772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40130)
0620 07:03:01.077081 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.077166 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 07:03:01.078274 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.077925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40140)
0620 07:03:01.078075 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.078153 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:03:01.078481 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:33829: Network error: Client connection negotiation failed: client connection to 127.0.97.61:33829: connect: Connection refused (error 111)
I20260620 07:03:01.086031 22854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:01.087421 22854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:01.088711 22854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:01.100332 22859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:01.100358 22862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:01.100406 22860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:01.100885 22854 server_base.cc:1034] running on GCE node
I20260620 07:03:01.101773 22854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:01.103242 22854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:01.104494 22854 hybrid_clock.cc:648] HybridClock initialized: now 1781938981104426 us; error 47 us; skew 500 ppm
I20260620 07:03:01.104841 22854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:01.107964 22854 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 07:03:01.109417 22854 webserver.cc:460] Webserver started at http://127.0.97.61:36279/ using document root <none> and password file <none>
I20260620 07:03:01.110121 22854 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:01.110224 22854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:01.110575 22854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:01.113656 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data/instance:
uuid: "b1be72dceeb74fbdbe89c9096a277c79"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "89d6f6d3d4f6cc6bc8381e895c900ae9"
server_key_iv: "dc6a1a9793c06b699d303db5c0093542"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.114490 22854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal/instance:
uuid: "b1be72dceeb74fbdbe89c9096a277c79"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "89d6f6d3d4f6cc6bc8381e895c900ae9"
server_key_iv: "dc6a1a9793c06b699d303db5c0093542"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.119748 22854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 07:03:01.123404 22869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:01.124975 22854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:01.125154 22854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "b1be72dceeb74fbdbe89c9096a277c79"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "89d6f6d3d4f6cc6bc8381e895c900ae9"
server_key_iv: "dc6a1a9793c06b699d303db5c0093542"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.125377 22854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:01.147997 22854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:01.152670 22854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:01.152951 22854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:01.171211 22854 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:33829
I20260620 07:03:01.171253 22920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:33829 every 8 connection(s)
I20260620 07:03:01.172511 22854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
I20260620 07:03:01.175462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22854
I20260620 07:03:01.175670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-1/wal/instance
I20260620 07:03:01.176211 388 external_mini_cluster.cc:1428] Setting key a3fcdcf9fedce641e21234a376ba20c3
I20260620 07:03:01.176594 22921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:01.185407 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 07:03:01.207652 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.188730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.97.62:40209)
0620 07:03:01.188925 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.188930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.190258 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 07:03:01.192243 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.192249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.192333 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.192443 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.194357 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.195381 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.197063 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.197684 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.197829 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.201570 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.201577 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.201582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.201628 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.204400 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.205260 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.205264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.205266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.205389 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.205852 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.205856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.205858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.206159 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.206281 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.207226 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.207409 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:03:01.208959 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.189205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40209 (local address 127.0.0.1:34400)
0620 07:03:01.190033 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.190090 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.191936 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.192734 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.192744 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.193512 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.194144 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.194164 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.195588 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.195593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.196884 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.196900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.197932 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.197937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.198150 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.198890 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.198914 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.201294 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.204535 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.204542 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.204555 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.205137 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.205499 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.205504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.205508 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.205736 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.206285 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.206293 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.206772 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.207072 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.207572 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260620 07:03:01.210736 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:01.226188 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.211502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40148)
0620 07:03:01.211740 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.211758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.211863 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.212891 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.212895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.213242 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.213372 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.213380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.215113 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.215119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.216191 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.216203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.217274 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.217279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.217415 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.218098 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.218116 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.220260 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.223531 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.223537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.223541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.224014 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.224417 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.224422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.224426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.224579 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.225223 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.225227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.225378 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.225515 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.225803 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 07:03:01.226294 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.211732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.0.97.61:33829)
0620 07:03:01.212417 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.212424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.212452 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:03:01.212500 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.212507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.212733 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.212866 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.213537 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.214953 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.216338 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.217125 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.217239 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.220481 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.220494 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.220508 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.220540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.223387 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.224175 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.224183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.224186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.224310 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.224725 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.224728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.224730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.225090 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.225259 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.225707 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.225947 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":375,"threads_started":1}
I20260620 07:03:01.228710 22921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:01.229938 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.229442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48324)
0620 07:03:01.229644 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.229787 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 07:03:01.230185 22870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.231685 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.231355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48338)
0620 07:03:01.231505 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.231581 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:03:01.231930 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 07:03:01.239796 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45125
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:03:01.281244 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 1
I20260620 07:03:01.282523 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.281991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48354)
0620 07:03:01.282249 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.282344 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 07:03:01.283737 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.283342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48356)
0620 07:03:01.283536 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.283615 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 07:03:01.283977 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.352216 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 2
I20260620 07:03:01.353433 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.352953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48364)
0620 07:03:01.353186 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.353276 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:03:01.354444 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.354086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48378)
0620 07:03:01.354258 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.354328 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 07:03:01.354704 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.377770 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } attempt: 5
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:01.392767 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.378585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40156)
0620 07:03:01.378818 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.378839 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.379068 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.379461 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.379465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.379844 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.380023 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.380032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.381419 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.381423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.382315 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.382327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.383350 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.383354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.383535 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.384202 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.384221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.386504 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.390349 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.390353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.390356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.390823 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.391227 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.391231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.391233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.391372 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.392011 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.392015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.392180 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.392297 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.392510 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 07:03:01.393009 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.378755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40156 (local address 127.0.97.61:33829)
0620 07:03:01.378993 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.379000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.379026 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:01.379204 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.379210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.379307 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.379414 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.380180 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.381246 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.382473 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.383170 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.383299 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.386772 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.386780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.386784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.386824 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.390172 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.390944 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.390950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.390954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.391099 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.391488 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.391493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.391496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.391868 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.392012 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.392445 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.392670 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 07:03:01.394076 22847 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:01.395164 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.394766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48392)
0620 07:03:01.394924 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.395001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:03:01.396087 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.395762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48394)
0620 07:03:01.395929 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.395995 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:03:01.396292 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.445951 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 3
I20260620 07:03:01.447225 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.446747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48400)
0620 07:03:01.446956 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.447048 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:03:01.447490 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 1
I20260620 07:03:01.448475 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.448086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48414)
0620 07:03:01.448255 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.448334 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 07:03:01.448477 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.447885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48402)
0620 07:03:01.448184 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.448292 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260620 07:03:01.448714 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.449790 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.449355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48424)
0620 07:03:01.449542 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.449638 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 07:03:01.450016 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.518283 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 2
I20260620 07:03:01.519568 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.519005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48434)
0620 07:03:01.519264 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.519375 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 07:03:01.520655 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.520260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48440)
0620 07:03:01.520422 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.520508 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 07:03:01.520898 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
W20260620 07:03:01.569882 22928 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:01.570231 22928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:01.570307 22928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:01.570389 22928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:01.577733 22928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:01.577848 22928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:01.577904 22928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:01.577953 22928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:01.578017 22928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:01.588189 22928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40209,127.0.97.61:33829,127.0.97.60:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:01.589519 22928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:01.590905 22928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 07:03:01.591980 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 4
I20260620 07:03:01.593182 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.592606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48456)
0620 07:03:01.592879 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.592986 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:03:01.594362 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.594026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48472)
0620 07:03:01.594178 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.594251 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:03:01.594626 22921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
W20260620 07:03:01.603361 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:01.603516 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:01.603411 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:01.604678 22928 server_base.cc:1034] running on GCE node
I20260620 07:03:01.605481 22928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:01.606936 22928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:01.608186 22928 hybrid_clock.cc:648] HybridClock initialized: now 1781938981608105 us; error 53 us; skew 500 ppm
I20260620 07:03:01.608517 22928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:01.616220 22928 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 07:03:01.617820 22928 webserver.cc:460] Webserver started at http://127.0.97.60:36131/ using document root <none> and password file <none>
I20260620 07:03:01.618505 22928 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:01.618640 22928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:01.618917 22928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:01.621862 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data/instance:
uuid: "54eca07037154cea93717a390b24c38f"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "81cbd6872cbb116a405938263d35bfd2"
server_key_iv: "a0c3f463ad4d72f86dba6e34ce04eccb"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.622750 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal/instance:
uuid: "54eca07037154cea93717a390b24c38f"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "81cbd6872cbb116a405938263d35bfd2"
server_key_iv: "a0c3f463ad4d72f86dba6e34ce04eccb"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.627228 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 3
I20260620 07:03:01.628216 22928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 07:03:01.628367 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.627809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48482)
0620 07:03:01.628054 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.628160 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 07:03:01.629593 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.629089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48498)
0620 07:03:01.629317 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.629421 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260620 07:03:01.629832 22847 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45125: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45125: connect: Connection refused (error 111)
I20260620 07:03:01.632288 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:01.633715 22928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 07:03:01.633865 22928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "54eca07037154cea93717a390b24c38f"
format_stamp: "Formatted at 2026-06-20 07:03:01 on dist-test-slave-mz20"
server_key: "81cbd6872cbb116a405938263d35bfd2"
server_key_iv: "a0c3f463ad4d72f86dba6e34ce04eccb"
server_key_version: "encryptionkey@0"
I20260620 07:03:01.634054 22928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:01.657439 22928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:01.663353 22928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:01.663669 22928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:01.684513 22928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45125
I20260620 07:03:01.684540 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45125 every 8 connection(s)
I20260620 07:03:01.686460 22928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
I20260620 07:03:01.691072 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:01.693944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 22928
I20260620 07:03:01.694139 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781938634772379-388-0/minicluster-data/master-2/wal/instance
I20260620 07:03:01.694692 388 external_mini_cluster.cc:1428] Setting key abe1fcad06913b406a73120c171f95f8
I20260620 07:03:01.699851 22995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:03:01.717779 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.702774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.97.62:40209)
0620 07:03:01.703004 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.703011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.703636 (+ 625us) server_negotiation.cc:407] Connection header received
0620 07:03:01.703805 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.703808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.703872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.703950 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.705283 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.706322 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.708162 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.708796 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.708936 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.712261 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.712266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.712269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.712304 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.714961 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.715667 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.715671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.715673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.715806 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.716195 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.716199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.716201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.716479 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.716622 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.717185 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.717403 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 07:03:01.718653 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.697307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.0.97.60:45125)
0620 07:03:01.698099 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.698112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.698162 (+ 50us) server_negotiation.cc:407] Connection header received
0620 07:03:01.700024 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.700046 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.700463 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.700784 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.702707 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.704694 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.706287 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.707349 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.707509 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.711865 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.711873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.711887 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.711921 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.714765 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.715458 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.715465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.715467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.715629 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.715991 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.715995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.715997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.716535 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.716817 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.717145 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.717477 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":585,"thread_start_us":345,"threads_started":1}
I20260620 07:03:01.718698 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.702740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40209 (local address 127.0.0.1:34416)
0620 07:03:01.703474 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.703523 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.703693 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.704040 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.704051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.704889 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.705127 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.705135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.706464 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.706469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.707982 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.707995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.709011 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.709017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.709142 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.709696 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.709717 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.712044 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.715109 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.715121 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.715135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.715531 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.715918 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.715923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.715926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.716078 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.716611 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.716619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.716826 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.717059 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.717778 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":300,"thread_start_us":341,"threads_started":1}
I20260620 07:03:01.720337 22995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:03:01.734689 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.721071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:33829 (local address 127.0.0.1:40162)
0620 07:03:01.721229 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.721246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.721341 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.721710 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.721715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.722149 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.722346 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.722358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.723417 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.723421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.724452 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.724474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.725152 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.725157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.725335 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.725972 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.725994 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.728558 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.732054 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.732059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.732063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.732580 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.732982 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.732987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.732991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.733164 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.733705 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.733710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.733855 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.734008 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.734297 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 07:03:01.734755 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.721101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.0.97.61:33829)
0620 07:03:01.721293 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.721298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.721316 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:01.721459 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.721466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.721546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.721648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.722498 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.723273 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.724603 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.725011 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.725103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.728799 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.728805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.728808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.728846 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.731888 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.732716 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.732720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.732722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.732862 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.733281 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.733285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.733287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.733564 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.733710 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.734109 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.734318 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 07:03:01.736123 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.721453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.97.62:40209)
0620 07:03:01.721632 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.721637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.721685 (+ 48us) server_negotiation.cc:407] Connection header received
0620 07:03:01.721846 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.721849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.721905 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.721974 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.723310 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.724026 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.725390 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.725903 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.725995 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.730431 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.730436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.730440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.730472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.733474 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.734145 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.734149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.734151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.734268 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.734634 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.734639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.734642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.734979 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.735269 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.735596 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.735813 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:03:01.736114 22995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:01.750746 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.736912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48514)
0620 07:03:01.737201 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.737214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.737308 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.737586 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.737590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.738056 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.738235 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.738246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.739675 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.739679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.740997 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.741010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.741879 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.741883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.742066 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.742620 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.742642 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.744958 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.748070 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.748076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.748080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.748615 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.748976 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.748979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.748981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.749139 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.749652 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.749657 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.749806 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.749979 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.750278 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 07:03:01.750756 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.736914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.97.60:45125)
0620 07:03:01.737096 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.737101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.737274 (+ 173us) server_negotiation.cc:407] Connection header received
0620 07:03:01.737409 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.737414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.737473 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.737548 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.738386 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.739506 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.741139 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.741739 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.741872 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.745179 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.745185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.745190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.745226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.747900 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.748733 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.748737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.748739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.748853 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.749254 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.749258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.749260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.749529 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.749633 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.750073 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.750407 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":49}
I20260620 07:03:01.753422 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.737774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.97.61:33829)
0620 07:03:01.737928 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.737933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.737949 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:03:01.738213 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.738219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.738287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.738383 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.739861 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.740894 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.742378 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.743017 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.743144 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.747454 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.747460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.747463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.747493 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.750463 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.751381 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.751386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.751389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.751528 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.752023 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.752025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.752027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.752359 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.752670 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.752879 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.753109 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 07:03:01.756525 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: Bootstrap starting.
I20260620 07:03:01.758401 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:01.759867 22995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:01.760943 22995 log.cc:824] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:01.763562 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: No bootstrap required, opened a new log
I20260620 07:03:01.761037 22822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "62862451b2b14ead80d3ff87ea5b57de"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34420
I20260620 07:03:01.770327 22995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.770699 22995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eca07037154cea93717a390b24c38f, State: Initialized, Role: FOLLOWER
I20260620 07:03:01.771131 22995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.772001 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:01.772372 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:01.773044 22995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: configured and running, proceeding with master startup.
W20260620 07:03:01.781946 23012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:03:01.782084 23012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:03:01.780431 22928 master_runner.cc:186] Error getting master registration for 127.0.97.62:40209: OK, instance_id { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" instance_seqno: 1781938980668444 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:03:01.791497 22928 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:03:01.807060 22847 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 4
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:03:01.823534 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.807767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48520)
0620 07:03:01.808063 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.808084 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.808207 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.808607 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.808612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.809121 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.809356 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.809370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.810919 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.810924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.812389 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.812404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.813424 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.813430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.813622 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.814240 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.814258 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.816574 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.820619 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.820625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.820630 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.821135 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.821529 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.821534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.821538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.821707 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.822475 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.822480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.822674 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.822818 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.823133 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 07:03:01.823540 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.807907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.0.97.60:45125)
0620 07:03:01.808149 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.808155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.808179 (+ 24us) server_negotiation.cc:407] Connection header received
0620 07:03:01.808347 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.808353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.808439 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.808561 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.809519 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.810777 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.812556 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.813275 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.813407 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.816798 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.816805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.816810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.816850 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.820466 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.821266 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.821272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.821275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.821421 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.821824 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.821829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.821833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.822361 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.822514 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.822943 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.823177 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 07:03:01.829728 22847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: Bootstrap starting.
I20260620 07:03:01.833753 22847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:01.835080 22847 log.cc:824] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:01.838598 22847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: No bootstrap required, opened a new log
I20260620 07:03:01.846172 22847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.846593 22847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62862451b2b14ead80d3ff87ea5b57de, State: Initialized, Role: FOLLOWER
I20260620 07:03:01.847105 22847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.848089 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:01.848448 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:01.849313 22847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:01.849367 22822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:03:01.849555 22822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:01.852250 22822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.854293 22970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62862451b2b14ead80d3ff87ea5b57de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "54eca07037154cea93717a390b24c38f"
I20260620 07:03:01.854825 22970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:01.858213 22970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62862451b2b14ead80d3ff87ea5b57de in term 1.
I20260620 07:03:01.858646 22822 leader_election.cc:288] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [CANDIDATE]: Term 1 election: Requested vote from peers b1be72dceeb74fbdbe89c9096a277c79 (127.0.97.61:33829), 54eca07037154cea93717a390b24c38f (127.0.97.60:45125)
I20260620 07:03:01.858980 22796 leader_election.cc:302] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54eca07037154cea93717a390b24c38f, 62862451b2b14ead80d3ff87ea5b57de; no voters:
I20260620 07:03:01.859470 23015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:01.872581 23015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [term 1 LEADER]: Becoming Leader. State: Replica: 62862451b2b14ead80d3ff87ea5b57de, State: Running, Role: LEADER
I20260620 07:03:01.873891 23015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.893790 22921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } attempt: 5
I20260620 07:03:01.903759 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62862451b2b14ead80d3ff87ea5b57de. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:01.904265 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: This master's current role is: LEADER
W20260620 07:03:01.908576 22801 tablet.cc:2369] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:01.910210 23035 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 07:03:01.918804 22949 tablet.cc:2369] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:01.930974 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.904870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.0.97.61:33829)
0620 07:03:01.905068 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.905074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.911026 (+ 5952us) server_negotiation.cc:407] Connection header received
0620 07:03:01.911287 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.911292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.911372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.911435 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.913671 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.914828 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.916783 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.917423 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.917527 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.920935 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.920942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.920947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.920988 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.924765 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.928445 (+ 3680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.928451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.928455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.928631 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.929053 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.929059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.929062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.929455 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.929764 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.930318 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.930610 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 07:03:01.932986 23036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:01.935770 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.885864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.0.97.62:40209)
0620 07:03:01.886169 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.886177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.906939 (+ 20762us) server_negotiation.cc:407] Connection header received
0620 07:03:01.908294 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.908301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.908388 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.908477 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.913228 (+ 4751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.914277 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.919121 (+ 4844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.919735 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.919835 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.927170 (+ 7335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.927178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.927183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.927226 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.931035 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.932740 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.932746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.932749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.932918 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.933573 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.933579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.933582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.933948 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.934222 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.935144 (+ 922us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.935402 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 07:03:01.939064 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.891183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.0.97.60:45125)
0620 07:03:01.891374 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.891380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.907825 (+ 16445us) server_negotiation.cc:407] Connection header received
0620 07:03:01.910628 (+ 2803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.910634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.910746 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.910822 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.916210 (+ 5388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.917296 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.920804 (+ 3508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.921468 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.921565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.924413 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.924420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.924425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.924466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.931546 (+ 7080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.936621 (+ 5075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.936626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.936629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.936803 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.937287 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.937292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.937294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.937623 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.937866 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.938410 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.938675 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 07:03:01.914749 22780 master_runner.cc:186] Error getting master registration for 127.0.97.61:33829: OK, instance_id { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" instance_seqno: 1781938981166190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:03:01.941757 22780 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 07:03:01 dist-test-slave-mz20 krb5kdc[22767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:03:01.946020 23036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:01.953755 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.894526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45125 (local address 127.0.0.1:48536)
0620 07:03:01.894872 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:01.894892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:01.895010 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:01.935773 (+ 40763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:01.935778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:01.936254 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:01.936454 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.936468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.940441 (+ 3973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.940446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.941777 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:01.941794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.942754 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.942760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:01.942952 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.943655 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:01.943677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:01.946210 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:01.950067 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.950072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.950076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.950603 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.952002 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:01.952007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:01.952010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:01.952182 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:01.952804 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:01.952809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:01.952955 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 07:03:01.953085 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 07:03:01.953387 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 07:03:01.953796 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:01.895447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.97.60:45125)
0620 07:03:01.935449 (+ 40002us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:01.935454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:01.935494 (+ 40us) server_negotiation.cc:407] Connection header received
0620 07:03:01.935545 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:01.935549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:01.935619 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:01.935717 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:01.937720 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.940312 (+ 2592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.941940 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:01.942620 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:01.942731 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:01.946582 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:01.946592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:01.946597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:01.946631 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:01.949913 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.951717 (+ 1804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.951721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.951725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.951862 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:01.952315 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:01.952319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:01.952322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:01.952701 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:01.952822 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:01.953197 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 07:03:01.953394 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39837,"thread_start_us":47360,"threads_started":1}
I20260620 07:03:01.960850 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: Bootstrap starting.
I20260620 07:03:01.965023 22921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:01.966334 22921 log.cc:824] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:01.973084 22970 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [term 1 FOLLOWER]: Refusing update from remote peer 62862451b2b14ead80d3ff87ea5b57de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:01.974301 23015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [LEADER]: Connected to new peer: Peer: permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:01.980326 22921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: No bootstrap required, opened a new log
I20260620 07:03:01.993224 22921 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.993827 22921 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1be72dceeb74fbdbe89c9096a277c79, State: Initialized, Role: FOLLOWER
I20260620 07:03:01.994781 22921 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } }
I20260620 07:03:01.996173 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:01.996634 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:01.998162 22895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:01.998242 22896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62862451b2b14ead80d3ff87ea5b57de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b1be72dceeb74fbdbe89c9096a277c79"
I20260620 07:03:02.001891 22895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [term 1 FOLLOWER]: Refusing update from remote peer 62862451b2b14ead80d3ff87ea5b57de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:02.005645 22896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62862451b2b14ead80d3ff87ea5b57de in term 1.
I20260620 07:03:02.006793 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62862451b2b14ead80d3ff87ea5b57de. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.007125 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:02.008876 23039 mvcc.cc:205] Tried to move back new op lower bound from 7298822070136745984 to 7298822069785067520. Current Snapshot: MvccSnapshot[applied={T|T < 7298822070136745984}]
I20260620 07:03:01.998162 22921 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:02.010408 23022 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:02.012610 23041 mvcc.cc:205] Tried to move back new op lower bound from 7298822070136745984 to 7298822069785067520. Current Snapshot: MvccSnapshot[applied={T|T < 7298822070136745984}]
I20260620 07:03:02.023187 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.023495 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:02.036352 23022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.036674 23022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:02.040167 23036 catalog_manager.cc:1347] Generated new cluster ID: 01793970fbfe4a748f003fd54bbf64d9
I20260620 07:03:02.040278 23036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:02.064842 23036 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:02.066929 23036 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:02.093439 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.093783 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:02.099704 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62862451b2b14ead80d3ff87ea5b57de. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.100031 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:02.107285 23061 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:03:02.108042 23052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "62862451b2b14ead80d3ff87ea5b57de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62862451b2b14ead80d3ff87ea5b57de" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40209 } } peers { permanent_uuid: "b1be72dceeb74fbdbe89c9096a277c79" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 33829 } } peers { permanent_uuid: "54eca07037154cea93717a390b24c38f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45125 } } }
I20260620 07:03:02.108422 23052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:02.117600 23036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62862451b2b14ead80d3ff87ea5b57de: Generated new TSK 0
I20260620 07:03:02.118846 23036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:03:02.220417 22849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.203638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34440 (local address 127.0.97.62:40209)
0620 07:03:02.203813 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.203818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.217469 (+ 13651us) server_negotiation.cc:407] Connection header received
0620 07:03:02.218057 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.218064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.218167 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.218246 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.220138 (+ 1892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34440: BlockingRecv error: recv got EOF from 127.0.0.1:34440 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 07:03:02.221695 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.203441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.0.97.61:33829)
0620 07:03:02.203633 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.203638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.219540 (+ 15902us) server_negotiation.cc:407] Connection header received
0620 07:03:02.219984 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.219989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.220088 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.220154 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.221477 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40194: BlockingRecv error: recv got EOF from 127.0.0.1:40194 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:03:02.222343 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.203819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.97.60:45125)
0620 07:03:02.203978 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.203984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.219139 (+ 15155us) server_negotiation.cc:407] Connection header received
0620 07:03:02.220881 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.220887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.220992 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.221073 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.222125 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48540: BlockingRecv error: recv got EOF from 127.0.0.1:48540 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 07:03:02.229717 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.227574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.0.97.61:33829)
0620 07:03:02.227790 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.227797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.228236 (+ 439us) server_negotiation.cc:407] Connection header received
0620 07:03:02.228481 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.228487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.228595 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.228681 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.229462 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40200: BlockingRecv error: recv got EOF from 127.0.0.1:40200 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 07:03:02.236248 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.233540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.97.61:33829)
0620 07:03:02.233701 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.233708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.235021 (+ 1313us) server_negotiation.cc:407] Connection header received
0620 07:03:02.235195 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.235201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.235273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.235354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.236045 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40210: BlockingRecv error: recv got EOF from 127.0.0.1:40210 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 07:03:02.242782 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.240874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.0.97.61:33829)
0620 07:03:02.241051 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.241057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.241555 (+ 498us) server_negotiation.cc:407] Connection header received
0620 07:03:02.241729 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.241734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.241806 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.241885 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.242558 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40224: BlockingRecv error: recv got EOF from 127.0.0.1:40224 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 07:03:02.251484 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.249424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.0.97.61:33829)
0620 07:03:02.249631 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.249637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.250132 (+ 495us) server_negotiation.cc:407] Connection header received
0620 07:03:02.250310 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.250316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.250390 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.250471 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.251309 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40230: BlockingRecv error: recv got EOF from 127.0.0.1:40230 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 07:03:02.266117 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.264097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.0.97.61:33829)
0620 07:03:02.264331 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.264338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.264801 (+ 463us) server_negotiation.cc:407] Connection header received
0620 07:03:02.264988 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.264992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.265061 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.265143 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.265891 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40234: BlockingRecv error: recv got EOF from 127.0.0.1:40234 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 07:03:02.294605 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.292465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.0.97.61:33829)
0620 07:03:02.292689 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.292697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.293229 (+ 532us) server_negotiation.cc:407] Connection header received
0620 07:03:02.293426 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.293431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.293505 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.293601 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.294336 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40250: BlockingRecv error: recv got EOF from 127.0.0.1:40250 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 07:03:02.332144 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.330136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.97.61:33829)
0620 07:03:02.330368 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.330374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.330701 (+ 327us) server_negotiation.cc:407] Connection header received
0620 07:03:02.330884 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.330890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.330996 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.331129 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.331888 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40254: BlockingRecv error: recv got EOF from 127.0.0.1:40254 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 07:03:02.401573 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.399577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.97.61:33829)
0620 07:03:02.399788 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.399795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.400126 (+ 331us) server_negotiation.cc:407] Connection header received
0620 07:03:02.400340 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.400347 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.400447 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.400575 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.401350 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40268: BlockingRecv error: recv got EOF from 127.0.0.1:40268 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 07:03:02.535266 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.533414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.97.61:33829)
0620 07:03:02.533619 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.533624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.533919 (+ 295us) server_negotiation.cc:407] Connection header received
0620 07:03:02.534062 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.534066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.534144 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.534240 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.535045 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40278: BlockingRecv error: recv got EOF from 127.0.0.1:40278 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 07:03:02.784178 23012 catalog_manager.cc:1259] Loaded cluster ID: 01793970fbfe4a748f003fd54bbf64d9
I20260620 07:03:02.784291 23012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: loading cluster ID for follower catalog manager: success
I20260620 07:03:02.786816 23012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: acquiring CA information for follower catalog manager: success
I20260620 07:03:02.788796 23012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54eca07037154cea93717a390b24c38f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:03:02.796242 22925 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:02.794608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.97.61:33829)
0620 07:03:02.794787 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:02.794792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:02.794989 (+ 197us) server_negotiation.cc:407] Connection header received
0620 07:03:02.795166 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:02.795170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:02.795242 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:02.795335 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:02.796008 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40294: BlockingRecv error: recv got EOF from 127.0.0.1:40294 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 07:03:03.111557 23061 catalog_manager.cc:1259] Loaded cluster ID: 01793970fbfe4a748f003fd54bbf64d9
I20260620 07:03:03.111670 23061 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: loading cluster ID for follower catalog manager: success
I20260620 07:03:03.114260 23061 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: acquiring CA information for follower catalog manager: success
I20260620 07:03:03.116045 23061 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b1be72dceeb74fbdbe89c9096a277c79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:03:03.319288 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:03.312260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.61:33829)
0620 07:03:03.312997 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:03.313002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:03.313021 (+ 19us) server_negotiation.cc:407] Connection header received
0620 07:03:03.313087 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:03.313090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:03.313250 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:03.313359 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:03.313889 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.314854 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.316830 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.317286 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.317390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:03.317947 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:03.318600 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:03.318723 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:03.318858 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 07:03:03.318942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":457,"threads_started":1}
I20260620 07:03:03.332126 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:03.326106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.97.60:45125)
0620 07:03:03.326796 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:03.326801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:03.326821 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:03:03.326916 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:03.326920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:03.327115 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:03.327260 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:03.327686 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.328471 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.329882 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.330385 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.330497 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:03.331014 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:03.331500 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:03.331630 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:03.331741 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 07:03:03.331824 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":478,"threads_started":1}
I20260620 07:03:03.364979 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:03.340691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.97.60:45125)
0620 07:03:03.340867 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:03.340873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:03.357950 (+ 17077us) server_negotiation.cc:407] Connection header received
0620 07:03:03.358129 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:03.358135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:03.358231 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:03.358343 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:03.359415 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.360510 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.362766 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.363374 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.363468 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:03.364173 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:03.364312 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:03.364386 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:03.364590 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 07:03:03.364678 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 07:03:03.365063 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:03.340140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.97.61:33829)
0620 07:03:03.340329 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:03.340336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:03.358609 (+ 18273us) server_negotiation.cc:407] Connection header received
0620 07:03:03.358889 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:03.358895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:03.358979 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:03.359038 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:03.359728 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.360814 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.362907 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.363577 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.363665 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:03.364269 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:03.364380 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:03.364451 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:03.364669 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 07:03:03.364754 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:03:03.365502 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:03.340557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.0.97.62:40209)
0620 07:03:03.356478 (+ 15921us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:03.356485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:03.356513 (+ 28us) server_negotiation.cc:407] Connection header received
0620 07:03:03.356656 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:03.356661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:03.356898 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:03.357117 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:03.359315 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.360428 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.362623 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:03.363232 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:03.363348 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:03.364084 (+ 736us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:03.364529 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:03.364622 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:03.365123 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 07:03:03.365209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15773,"thread_start_us":15670,"threads_started":1}
I20260620 07:03:03.370599 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22780
I20260620 07:03:03.371672 22842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:03.521116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22780
W20260620 07:03:03.538112 22944 connection.cc:535] server connection from 127.0.0.1:48520 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 07:03:03.538730 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22854
I20260620 07:03:03.539876 22916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:03.646205 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22854
I20260620 07:03:03.662498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 22928
I20260620 07:03:03.663897 22990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:03.770309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 22928
2026-06-20T07:03:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4839 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 07:03:03 dist-test-slave-mz20 krb5kdc[23147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 07:03:03 dist-test-slave-mz20 krb5kdc[23147](info): set up 2 sockets
Jun 20 07:03:03 dist-test-slave-mz20 krb5kdc[23147](info): commencing operation
krb5kdc: starting...
W20260620 07:03:04.827423 388 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/test-user.keytab.
Jun 20 07:03:05 dist-test-slave-mz20 krb5kdc[23147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T07:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T07:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 07:03:05.126099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41045
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 07:03:05.508620 23160 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:05.509012 23160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:05.509091 23160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:05.509177 23160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:05.517205 23160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:05.517320 23160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:05.517380 23160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:05.517431 23160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:05.517496 23160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:05.528237 23160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:05.529784 23160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:05.531332 23160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:05.544370 23165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:05.544348 23166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:05.544426 23168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:05.545151 23160 server_base.cc:1034] running on GCE node
I20260620 07:03:05.546028 23160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:05.547791 23160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:05.549060 23160 hybrid_clock.cc:648] HybridClock initialized: now 1781938985548984 us; error 36 us; skew 500 ppm
I20260620 07:03:05.549440 23160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:05 dist-test-slave-mz20 krb5kdc[23147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:05.553380 23160 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 07:03:05.555277 23160 webserver.cc:460] Webserver started at http://127.0.97.62:38755/ using document root <none> and password file <none>
I20260620 07:03:05.556149 23160 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:05.556259 23160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:05.556627 23160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:05.560264 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data/instance:
uuid: "4c4fadb513594da89109a8c8a33a8b5c"
format_stamp: "Formatted at 2026-06-20 07:03:05 on dist-test-slave-mz20"
server_key: "12a03024293c64120d4605c3e508e9ad"
server_key_iv: "cfba6deff7c6d8dcb885a3ff100c59e6"
server_key_version: "encryptionkey@0"
I20260620 07:03:05.561178 23160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c4fadb513594da89109a8c8a33a8b5c"
format_stamp: "Formatted at 2026-06-20 07:03:05 on dist-test-slave-mz20"
server_key: "12a03024293c64120d4605c3e508e9ad"
server_key_iv: "cfba6deff7c6d8dcb885a3ff100c59e6"
server_key_version: "encryptionkey@0"
I20260620 07:03:05.567334 23160 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 07:03:05.571311 23175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:05.573001 23160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 07:03:05.573217 23160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal
uuid: "4c4fadb513594da89109a8c8a33a8b5c"
format_stamp: "Formatted at 2026-06-20 07:03:05 on dist-test-slave-mz20"
server_key: "12a03024293c64120d4605c3e508e9ad"
server_key_iv: "cfba6deff7c6d8dcb885a3ff100c59e6"
server_key_version: "encryptionkey@0"
I20260620 07:03:05.573473 23160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:05.604665 23160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:05.614233 23160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:05.614590 23160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:05.635166 23160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41045
I20260620 07:03:05.635190 23226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41045 every 8 connection(s)
I20260620 07:03:05.636775 23160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/data/info.pb
I20260620 07:03:05.641330 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:05.646575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 23160
I20260620 07:03:05.646792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-0/wal/instance
I20260620 07:03:05.647413 388 external_mini_cluster.cc:1428] Setting key 388a1a0e03164e38276c2fe9cf22c387
I20260620 07:03:05.650043 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:05 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:03:05.677193 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.655617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56298 (local address 127.0.97.62:41045)
0620 07:03:05.656347 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:05.656358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:05.656412 (+ 54us) server_negotiation.cc:407] Connection header received
0620 07:03:05.657895 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:05.657904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:05.658259 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:05.658491 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:05.659860 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:05.661474 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:05.663599 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:05.664326 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:05.664509 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:05.668840 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:05.668849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:05.668866 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:05.668905 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:05.671897 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:05.672813 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:05.672821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:05.672823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:05.673007 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:05.673512 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:05.673517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:05.673519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:05.674200 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:05.674425 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:05.675173 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 07:03:05.675874 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":363,"server-negotiator.queue_time_us":500,"thread_start_us":338,"threads_started":1}
I20260620 07:03:05.677254 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.654090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41045 (local address 127.0.0.1:56298)
0620 07:03:05.655078 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.655145 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:05.657445 (+ 2300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:05.658533 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:05.658566 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:05.659215 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:05.659624 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:05.659643 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:05.661658 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:05.661663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:05.663424 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:05.663439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:05.664600 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:05.664606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:05.664796 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:05.665565 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:05.665594 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:05.668567 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:05.672041 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:05.672054 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:05.672069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:05.672638 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:05.673128 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:05.673132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:05.673136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:05.673364 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:05.674354 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:05.674363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:05.674680 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 07:03:05.674994 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 07:03:05.675720 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":4,"thread_start_us":438,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 07:03:05.681203 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 07:03:05.682777 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.682099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40760)
0620 07:03:05.682322 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.682598 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 07:03:05.683126 23176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
I20260620 07:03:05.684855 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.684458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40762)
0620 07:03:05.684627 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.684713 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 07:03:05.685075 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41691: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 07:03:05.718636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41691
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:03:05.734525 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } attempt: 1
I20260620 07:03:05.736140 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.735534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40774)
0620 07:03:05.735802 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.735937 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 07:03:05.737260 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.736863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40784)
0620 07:03:05.737018 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.737098 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 07:03:05.737471 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41691: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
I20260620 07:03:05.805815 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } attempt: 2
I20260620 07:03:05.807564 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.806894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40788)
0620 07:03:05.807207 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.807353 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 07:03:05.808820 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.808362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40792)
0620 07:03:05.808539 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.808650 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 07:03:05.809055 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41691: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
I20260620 07:03:05.900359 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } attempt: 3
I20260620 07:03:05.901939 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.901304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40804)
0620 07:03:05.901584 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.901725 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 07:03:05.903162 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:05.902694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40812)
0620 07:03:05.902861 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:05.902976 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:03:05.903401 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41691: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
I20260620 07:03:06.046710 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } attempt: 4
I20260620 07:03:06.048013 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.047352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40828)
0620 07:03:06.047733 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.047846 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260620 07:03:06.049129 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.048724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40830)
0620 07:03:06.048880 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.048980 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 07:03:06.049358 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:41691: Network error: Client connection negotiation failed: client connection to 127.0.97.61:41691: connect: Connection refused (error 111)
W20260620 07:03:06.082698 23234 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:06.083091 23234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:06.083164 23234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:06.083251 23234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:06.090526 23234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:06.090651 23234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:06.090710 23234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 07:03:06.090758 23234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:06.090816 23234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:06.100808 23234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:06.102138 23234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:06.103564 23234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:06.115360 23239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:06.115391 23240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:06.115444 23242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:06.116135 23234 server_base.cc:1034] running on GCE node
I20260620 07:03:06.116904 23234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:06.118433 23234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:06.119676 23234 hybrid_clock.cc:648] HybridClock initialized: now 1781938986119605 us; error 41 us; skew 500 ppm
I20260620 07:03:06.120014 23234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:06.123258 23234 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 07:03:06.124666 23234 webserver.cc:460] Webserver started at http://127.0.97.61:39897/ using document root <none> and password file <none>
I20260620 07:03:06.125371 23234 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:06.125468 23234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:06.125773 23234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:06.128803 23234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data/instance:
uuid: "65b7430b2c824ac98c85528148e48413"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "3dd5bc70af5f8588d59909665dbfc3c0"
server_key_iv: "cf676ebf390d8a3e5c26e42891461f5f"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.129577 23234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal/instance:
uuid: "65b7430b2c824ac98c85528148e48413"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "3dd5bc70af5f8588d59909665dbfc3c0"
server_key_iv: "cf676ebf390d8a3e5c26e42891461f5f"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.135103 23234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 07:03:06.139083 23249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:06.140640 23234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 07:03:06.140834 23234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal
uuid: "65b7430b2c824ac98c85528148e48413"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "3dd5bc70af5f8588d59909665dbfc3c0"
server_key_iv: "cf676ebf390d8a3e5c26e42891461f5f"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.141060 23234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:06.161108 23234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:06.165401 23234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:06.165699 23234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:06.183125 23234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41691
I20260620 07:03:06.183168 23300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41691 every 8 connection(s)
I20260620 07:03:06.184515 23234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/data/info.pb
I20260620 07:03:06.188330 23301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:06.189466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 23234
I20260620 07:03:06.189683 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-1/wal/instance
I20260620 07:03:06.190335 388 external_mini_cluster.cc:1428] Setting key 17ff965a8575afa2ffb3234c7795e9ea
I20260620 07:03:06.196836 23301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 07:03:06.219623 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.200176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.0.97.62:41045)
0620 07:03:06.202086 (+ 1910us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.202094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.202800 (+ 706us) server_negotiation.cc:407] Connection header received
0620 07:03:06.204765 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.204771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.204864 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.204961 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.206592 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.207375 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.208870 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.209296 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.209443 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.213058 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.213064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.213067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.213107 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.216135 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.216975 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.216981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.216985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.217119 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.217580 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.217584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.217587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.217945 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.218099 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.218990 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.219237 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1753,"thread_start_us":1613,"threads_started":1}
I20260620 07:03:06.220610 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.200744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41045 (local address 127.0.0.1:56310)
0620 07:03:06.202628 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.202676 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.204463 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.205239 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.205247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.205776 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.206342 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.206359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.207544 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.207548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.208717 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.208729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.209534 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.209541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.209775 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.210424 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.210447 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.212781 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.216314 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.216325 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.216345 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.216835 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.217243 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.217249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.217253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.217457 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.218099 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.218109 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.218573 (+ 464us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.218885 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.219436 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":1295,"threads_started":1}
I20260620 07:03:06.222513 23301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:06.236953 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.223287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40832)
0620 07:03:06.223486 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.223504 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.223609 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.224436 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.224440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.224871 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.225038 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.225048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.226405 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.226410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.227352 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.227365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.228302 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.228308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.228478 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.229021 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.229039 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.231265 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.234514 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.234520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.234525 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.234933 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.235313 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.235318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.235320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.235432 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.236025 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.236030 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.236181 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.236288 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.236568 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 07:03:06.237231 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.223428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.97.61:41691)
0620 07:03:06.223998 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.224005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.224031 (+ 26us) server_negotiation.cc:407] Connection header received
0620 07:03:06.224076 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.224081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.224261 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.224377 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.225194 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.226229 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.227547 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.228160 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.228275 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.231523 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.231536 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.231552 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.231582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.234357 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.235070 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.235077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.235080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.235190 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.235544 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.235548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.235550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.235895 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.236086 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.236502 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.236726 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":277,"threads_started":1}
I20260620 07:03:06.239862 23301 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:06.241050 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.240576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41032)
0620 07:03:06.240756 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.240870 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 07:03:06.241317 23252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.242967 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.242592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41038)
0620 07:03:06.242751 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.242851 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 07:03:06.243176 23301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 07:03:06.249720 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8UMGcc/build/asan/bin/kudu
/tmp/dist-test-task8UMGcc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35357
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 07:03:06.292495 23301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 1
I20260620 07:03:06.293596 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.293157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41054)
0620 07:03:06.293358 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.293443 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 07:03:06.294736 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.294324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41068)
0620 07:03:06.294492 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.294625 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 07:03:06.294956 23301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.348634 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } attempt: 5
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:06.362267 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.349442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40834)
0620 07:03:06.349651 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.349673 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.349863 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.350264 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.350269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.350735 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.350932 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.350942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.351952 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.351956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.352784 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.352807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.353571 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.353577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.353774 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.354499 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.354519 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.356858 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.359895 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.359901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.359906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.360273 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.360647 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.360652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.360655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.360786 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.361311 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.361316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.361459 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.361592 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.361911 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 07:03:06.362267 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.349572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.0.97.61:41691)
0620 07:03:06.349741 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.349747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.349769 (+ 22us) server_negotiation.cc:407] Connection header received
0620 07:03:06.350012 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.350016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.350107 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.350194 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.351097 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.351787 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.352953 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.353412 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.353526 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.357112 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.357117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.357121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.357156 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.359747 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.360400 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.360404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.360406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.360532 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.360896 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.360900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.360902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.361190 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.361320 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.361714 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.361926 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 07:03:06.363178 23301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 2
I20260620 07:03:06.363516 23227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:06.364656 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.364239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41074)
0620 07:03:06.364434 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.364527 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 07:03:06.365001 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.364279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41072)
0620 07:03:06.364529 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.364621 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260620 07:03:06.365633 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.365214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41084)
0620 07:03:06.365392 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.365472 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 07:03:06.365942 23301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.366102 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.365730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41094)
0620 07:03:06.365891 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.365972 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 07:03:06.366294 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.417497 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 1
I20260620 07:03:06.418877 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.418284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41106)
0620 07:03:06.418531 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.418672 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 07:03:06.419942 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.419557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41108)
0620 07:03:06.419725 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.419811 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 07:03:06.420171 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.457180 23301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 3
I20260620 07:03:06.458451 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.457829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41118)
0620 07:03:06.458103 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.458225 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 07:03:06.459695 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.459197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41122)
0620 07:03:06.459454 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.459550 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260620 07:03:06.459940 23301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.488431 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 2
I20260620 07:03:06.489653 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.489117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41138)
0620 07:03:06.489362 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.489476 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 07:03:06.490691 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.490331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41140)
0620 07:03:06.490493 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.490583 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 07:03:06.490902 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
W20260620 07:03:06.589123 23309 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 07:03:06.589458 23309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 07:03:06.589558 23309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 07:03:06.589674 23309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 07:03:06.597092 23309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 07:03:06.597193 23309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 07:03:06.597259 23309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260620 07:03:06.597138 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 3
W20260620 07:03:06.597328 23309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 07:03:06.597414 23309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 07:03:06.598227 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.597766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41142)
0620 07:03:06.597983 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.598073 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 07:03:06.599339 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.598948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41154)
0620 07:03:06.599159 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.599232 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 07:03:06.599548 23227 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.603168 23301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 4
I20260620 07:03:06.604207 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.603720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41158)
0620 07:03:06.603936 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.604029 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 07:03:06.605227 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.604871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41174)
0620 07:03:06.605028 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.605107 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 07:03:06.605450 23301 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:35357: Network error: Client connection negotiation failed: client connection to 127.0.97.60:35357: connect: Connection refused (error 111)
I20260620 07:03:06.608839 23309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41045,127.0.97.61:41691,127.0.97.60:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 06:01:36 UTC on ccycloud.quasar-ouulox.root.comops.site
build id 4457022
ASAN enabled
I20260620 07:03:06.610080 23309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 07:03:06.611486 23309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 07:03:06.622891 23314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:06.623160 23317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 07:03:06.623234 23315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 07:03:06.624258 23309 server_base.cc:1034] running on GCE node
I20260620 07:03:06.625072 23309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 07:03:06.626634 23309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 07:03:06.627905 23309 hybrid_clock.cc:648] HybridClock initialized: now 1781938986627818 us; error 59 us; skew 500 ppm
I20260620 07:03:06.628244 23309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 07:03:06.631371 23309 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 07:03:06.632817 23309 webserver.cc:460] Webserver started at http://127.0.97.60:37769/ using document root <none> and password file <none>
I20260620 07:03:06.633711 23309 fs_manager.cc:359] Metadata directory not provided
I20260620 07:03:06.633801 23309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 07:03:06.634052 23309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 07:03:06.637065 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data/instance:
uuid: "867efa4ee06b48ec875110aba103eb1b"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "a16fd39fa744e6ca8983023796c49751"
server_key_iv: "4453211799fe077031a71a743d13cef4"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.637866 23309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal/instance:
uuid: "867efa4ee06b48ec875110aba103eb1b"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "a16fd39fa744e6ca8983023796c49751"
server_key_iv: "4453211799fe077031a71a743d13cef4"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.643357 23309 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 07:03:06.646884 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 07:03:06.648234 23309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 07:03:06.648419 23309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data,/tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal
uuid: "867efa4ee06b48ec875110aba103eb1b"
format_stamp: "Formatted at 2026-06-20 07:03:06 on dist-test-slave-mz20"
server_key: "a16fd39fa744e6ca8983023796c49751"
server_key_iv: "4453211799fe077031a71a743d13cef4"
server_key_version: "encryptionkey@0"
I20260620 07:03:06.648618 23309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 07:03:06.663023 23309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 07:03:06.667294 23309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 07:03:06.667550 23309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 07:03:06.685294 23309 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:35357
I20260620 07:03:06.685309 23375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:35357 every 8 connection(s)
I20260620 07:03:06.686985 23309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/data/info.pb
I20260620 07:03:06.690908 23376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 07:03:06.694052 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8UMGcc/build/asan/bin/kudu as pid 23309
I20260620 07:03:06.694207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8UMGcc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781938634772379-388-0/minicluster-data/master-2/wal/instance
I20260620 07:03:06.694715 388 external_mini_cluster.cc:1428] Setting key 8b45f9b58d6ecce0a3a9281dbceebd7b
I20260620 07:03:06.698796 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:03:06.717484 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.702003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.97.62:41045)
0620 07:03:06.702219 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.702226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.703033 (+ 807us) server_negotiation.cc:407] Connection header received
0620 07:03:06.703202 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.703207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.703320 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.703428 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.704892 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.705765 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.707725 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.708195 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.708303 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.712040 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.712046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.712050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.712086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.714530 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.715387 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.715393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.715396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.715524 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.715889 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.715894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.715897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.716162 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.716297 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.716875 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.717105 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 07:03:06.718334 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.701983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41045 (local address 127.0.0.1:56324)
0620 07:03:06.702883 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.702925 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.703085 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.703502 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.703507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.704351 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.704678 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.704692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.706068 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.706072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.707524 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.707542 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.708467 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.708475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.708701 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.709342 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.709381 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.711778 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.714788 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.714798 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.714809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.715234 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.715637 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.715642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.715645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.715769 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.716294 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.716306 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.716513 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.716724 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.717433 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":208,"thread_start_us":419,"threads_started":1}
I20260620 07:03:06.718313 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.696646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.0.97.60:35357)
0620 07:03:06.697456 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.697468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.697510 (+ 42us) server_negotiation.cc:407] Connection header received
0620 07:03:06.699314 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.699339 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.699814 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.700206 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.701868 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.703552 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.705336 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.706435 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.706653 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.711166 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.711176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.711191 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.711225 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.714196 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.715040 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.715047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.715049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.715217 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.715584 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.715587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.715589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.716133 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.716443 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.716754 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.717145 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260620 07:03:06.720750 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 07:03:06.733780 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.721626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.0.97.61:41691)
0620 07:03:06.721797 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.721803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.721821 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:06.721996 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.722001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.722062 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.722139 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.722948 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.723602 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.724772 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.725185 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.725294 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.728501 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.728508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.728513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.728553 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.731163 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.731872 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.731876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.731878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.732000 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.732375 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.732379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.732383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.732707 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.732864 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.733243 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.733465 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 07:03:06.733803 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.721514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41691 (local address 127.0.0.1:40844)
0620 07:03:06.721763 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.721780 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.721868 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.722182 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.722187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.722624 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.722793 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.722802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.723752 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.723758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.724627 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.724638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.725335 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.725341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.725559 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.726115 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.726132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.728301 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.731289 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.731295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.731299 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.731754 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.732108 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.732113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.732117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.732277 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.732854 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.732858 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.733003 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.733173 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.733436 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260620 07:03:06.734771 23376 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 07:03:06.735118 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.721311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.97.62:41045)
0620 07:03:06.721610 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.721616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.721633 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:06.721682 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.721687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.721767 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.721855 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.723089 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.723750 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.724893 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.725312 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.725417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.729530 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.729535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.729538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.729569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.732314 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.733061 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.733065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.733069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.733238 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.733620 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.733625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.733629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.733974 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.734279 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.734673 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.734837 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 07:03:06.748889 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.735488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41196)
0620 07:03:06.735645 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.735663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.735752 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.736139 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.736143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.736558 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.736723 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.736735 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.737986 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.737989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.739192 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.739207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.740069 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.740073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.740238 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.740793 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.740814 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.743175 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.746261 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.746266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.746269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.746722 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.747084 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.747089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.747093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.747248 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.747888 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.747894 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.748047 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.748203 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.748490 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 07:03:06.748915 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.735702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.97.60:35357)
0620 07:03:06.735862 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.735868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.735886 (+ 18us) server_negotiation.cc:407] Connection header received
0620 07:03:06.735928 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.735932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.736000 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.736094 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.736848 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.737845 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.739336 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.739910 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.740034 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.743361 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.743365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.743369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.743395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.746146 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.746852 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.746855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.746858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.746969 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.747402 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.747407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.747411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.747724 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.747836 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.748306 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.748666 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":55}
I20260620 07:03:06.751458 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.736716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.97.61:41691)
0620 07:03:06.736874 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.736878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.736891 (+ 13us) server_negotiation.cc:407] Connection header received
0620 07:03:06.736992 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.736996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.737045 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.737111 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.738424 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.739421 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.740860 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.741500 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.741623 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.745764 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.745768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.745772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.745800 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.748850 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.749587 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.749590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.749593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.749712 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.750089 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.750094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.750096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.750394 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.750674 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.750959 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.751164 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:03:06.754117 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: Bootstrap starting.
I20260620 07:03:06.755986 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 07:03:06.757293 23376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:06.758512 23376 log.cc:824] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:06.761591 23376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: No bootstrap required, opened a new log
I20260620 07:03:06.758308 23202 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4c4fadb513594da89109a8c8a33a8b5c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56338
I20260620 07:03:06.772518 23376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.773036 23376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867efa4ee06b48ec875110aba103eb1b, State: Initialized, Role: FOLLOWER
I20260620 07:03:06.773546 23376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.774492 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.774888 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:06.775664 23376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:06.776775 23227 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 4
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 07:03:06.786733 23393 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 07:03:06.786870 23393 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:03:06.795070 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.777495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41200)
0620 07:03:06.777667 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.777696 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.777825 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.778259 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.778264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.778829 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.778986 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.778995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.780614 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.780620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.782071 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.782087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.783176 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.783183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.783418 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.784162 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.784186 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.787083 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.791746 (+ 4663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.791753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.791757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.792477 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.792974 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.792978 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.792982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.793167 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.793917 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.793921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.794092 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.794238 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.794688 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 07:03:06.795084 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.777583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.0.97.60:35357)
0620 07:03:06.777798 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.777804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.777829 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:06.778012 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.778018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.778093 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.778189 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.779144 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.780391 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.782230 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.782981 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.783102 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.787412 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.787419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.787425 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.787472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.791555 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.792654 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.792660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.792664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.792825 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.793290 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.793303 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.793306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.793777 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.793953 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.794373 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.794698 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 07:03:06.784657 23309 master_runner.cc:186] Error getting master registration for 127.0.97.62:41045: OK, instance_id { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" instance_seqno: 1781938985629301 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:03:06.795728 23309 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:03:06.802239 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: Bootstrap starting.
I20260620 07:03:06.806910 23227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:06.808331 23227 log.cc:824] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:06.812156 23227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: No bootstrap required, opened a new log
I20260620 07:03:06.820039 23227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.820515 23227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4fadb513594da89109a8c8a33a8b5c, State: Initialized, Role: FOLLOWER
I20260620 07:03:06.821045 23227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.822088 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.822469 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:06.823452 23227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:06.823558 23202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 07:03:06.823755 23202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:06.826303 23202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.828073 23202 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [CANDIDATE]: Term 1 election: Requested vote from peers 65b7430b2c824ac98c85528148e48413 (127.0.97.61:41691), 867efa4ee06b48ec875110aba103eb1b (127.0.97.60:35357)
I20260620 07:03:06.828444 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c4fadb513594da89109a8c8a33a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "867efa4ee06b48ec875110aba103eb1b"
I20260620 07:03:06.828976 23351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:06.831480 23351 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c4fadb513594da89109a8c8a33a8b5c in term 1.
I20260620 07:03:06.832167 23178 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4fadb513594da89109a8c8a33a8b5c, 867efa4ee06b48ec875110aba103eb1b; no voters:
I20260620 07:03:06.832664 23396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 07:03:06.861068 23396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [term 1 LEADER]: Becoming Leader. State: Replica: 4c4fadb513594da89109a8c8a33a8b5c, State: Running, Role: LEADER
I20260620 07:03:06.861789 23396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
W20260620 07:03:06.867926 23181 tablet.cc:2369] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:06.874979 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4fadb513594da89109a8c8a33a8b5c. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.875432 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:06.881883 23414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 07:03:06.899916 23414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 07:03:06.904640 23301 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } attempt: 5
W20260620 07:03:06.919627 23330 tablet.cc:2369] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 07:03:06.921257 23420 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 07:03:06.924860 23351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [term 1 FOLLOWER]: Refusing update from remote peer 4c4fadb513594da89109a8c8a33a8b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:06.925840 23401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:06.933360 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.853841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.97.62:41045)
0620 07:03:06.854051 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.854057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.856267 (+ 2210us) server_negotiation.cc:407] Connection header received
0620 07:03:06.857252 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.857258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.857350 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.857429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.873030 (+ 15601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.874103 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.886341 (+ 12238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.887158 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.887279 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.909756 (+ 22477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.909764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.909769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.909815 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.913797 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.917567 (+ 3770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.917573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.917576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.917747 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.919657 (+ 1910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.919662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.919665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.920067 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.920337 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.932418 (+ 12081us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.932699 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 07:03:06.927095 23160 master_runner.cc:186] Error getting master registration for 127.0.97.61:41691: OK, instance_id { permanent_uuid: "65b7430b2c824ac98c85528148e48413" instance_seqno: 1781938986178253 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 07:03:06.940202 23160 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 07:03:06.942317 23306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.855650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.0.97.61:41691)
0620 07:03:06.855837 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.855844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.875024 (+ 19180us) server_negotiation.cc:407] Connection header received
0620 07:03:06.876850 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.876857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.876961 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.877051 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.889312 (+ 12261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.890421 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.901057 (+ 10636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.901712 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.901824 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.922219 (+ 20395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.922227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.922231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.922274 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.926504 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.938768 (+ 12264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.938774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.938779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.938978 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.940686 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.940691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.940695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.941057 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.941337 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.941693 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.941950 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 07:03:06.943965 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.857944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.0.97.60:35357)
0620 07:03:06.858132 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.858138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.878291 (+ 20153us) server_negotiation.cc:407] Connection header received
0620 07:03:06.879453 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.879460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.879560 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.879661 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.891753 (+ 12092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.892901 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.902522 (+ 9621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.903241 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.903350 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.927137 (+ 23787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.927144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.927149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.927191 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.931237 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.939251 (+ 8014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.939257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.939260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.939557 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.940489 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.940495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.940498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.942053 (+ 1555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.942315 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.943343 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.943594 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 07:03:06 dist-test-slave-mz20 krb5kdc[23147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781938986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 07:03:06.955987 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4fadb513594da89109a8c8a33a8b5c. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.956357 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:06.959456 23418 mvcc.cc:205] Tried to move back new op lower bound from 7298822090410946560 to 7298822090192695296. Current Snapshot: MvccSnapshot[applied={T|T < 7298822090410946560}]
I20260620 07:03:06.959553 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.905234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:35357 (local address 127.0.0.1:41210)
0620 07:03:06.905596 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 07:03:06.905615 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 07:03:06.905732 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 07:03:06.935106 (+ 29374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 07:03:06.935112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 07:03:06.935618 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 07:03:06.935833 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.935846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.939934 (+ 4088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.939940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.941295 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 07:03:06.941309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.945261 (+ 3952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.945268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 07:03:06.945487 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.946271 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 07:03:06.946293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 07:03:06.949047 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 07:03:06.955651 (+ 6604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.955657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.955662 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.956219 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.956885 (+ 666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 07:03:06.956891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 07:03:06.956895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 07:03:06.957065 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 07:03:06.958491 (+ 1426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 07:03:06.958496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 07:03:06.958691 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 07:03:06.958831 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 07:03:06.959152 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260620 07:03:06.964049 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:06.905942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.0.97.60:35357)
0620 07:03:06.923432 (+ 17490us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:06.923438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:06.923463 (+ 25us) server_negotiation.cc:407] Connection header received
0620 07:03:06.923522 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:06.923527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:06.923607 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:06.936027 (+ 12420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:06.936320 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.937598 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.941516 (+ 3918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:06.942574 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:06.945900 (+ 3326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:06.949433 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 07:03:06.949442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 07:03:06.949478 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 07:03:06.949550 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 07:03:06.955390 (+ 5840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.956444 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.956450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.956453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.956698 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 07:03:06.957257 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 07:03:06.957263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 07:03:06.957266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 07:03:06.958299 (+ 1033us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 07:03:06.960218 (+ 1919us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:06.960426 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 07:03:06.960802 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17320,"thread_start_us":17754,"threads_started":1}
I20260620 07:03:06.970245 23301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: Bootstrap starting.
I20260620 07:03:06.973489 23382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.973769 23382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:06.978508 23401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.978852 23401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:06.979837 23301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: Neither blocks nor log segments found. Creating new log.
I20260620 07:03:06.981958 23301 log.cc:824] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: Log is configured to *not* fsync() on all Append() calls
I20260620 07:03:06.983919 23414 catalog_manager.cc:1347] Generated new cluster ID: 404f044d482541dc9d8ff423b1bb600e
I20260620 07:03:06.984025 23414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 07:03:06.985880 23301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: No bootstrap required, opened a new log
I20260620 07:03:06.996363 23301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.996742 23301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b7430b2c824ac98c85528148e48413, State: Initialized, Role: FOLLOWER
I20260620 07:03:06.997218 23301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } }
I20260620 07:03:06.998579 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:06.999419 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:06.999537 23301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 07:03:06.999611 23276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c4fadb513594da89109a8c8a33a8b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65b7430b2c824ac98c85528148e48413"
I20260620 07:03:07.000201 23275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [term 0 FOLLOWER]: Advancing to term 1
I20260620 07:03:07.004359 23275 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [term 1 FOLLOWER]: Refusing update from remote peer 4c4fadb513594da89109a8c8a33a8b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 07:03:07.006292 23396 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 07:03:07.050909 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4fadb513594da89109a8c8a33a8b5c. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:07.051249 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:07.052237 23276 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c4fadb513594da89109a8c8a33a8b5c for term 1 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 07:03:07.051802 23437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:07.058063 23437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 07:03:07.084923 23396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c4fadb513594da89109a8c8a33a8b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c4fadb513594da89109a8c8a33a8b5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41045 } } peers { permanent_uuid: "65b7430b2c824ac98c85528148e48413" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41691 } } peers { permanent_uuid: "867efa4ee06b48ec875110aba103eb1b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 35357 } } }
I20260620 07:03:07.085302 23396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c [sys.catalog]: This master's current role is: LEADER
I20260620 07:03:07.088457 23442 catalog_manager.cc:1259] Loaded cluster ID: 404f044d482541dc9d8ff423b1bb600e
I20260620 07:03:07.088635 23442 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: loading cluster ID for follower catalog manager: success
W20260620 07:03:07.090889 23442 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 07:03:07.091117 23442 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 07:03:07.092409 23414 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 07:03:07.094305 23414 catalog_manager.cc:1504] Loading token signing keys...
I20260620 07:03:07.132303 23414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4fadb513594da89109a8c8a33a8b5c: Generated new TSK 0
I20260620 07:03:07.133798 23414 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 07:03:07.287838 23303 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.255118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.0.97.62:41045)
0620 07:03:07.255308 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.255314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.284296 (+ 28982us) server_negotiation.cc:407] Connection header received
0620 07:03:07.286291 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.286298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.286386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.286454 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.287563 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56356: BlockingRecv error: recv got EOF from 127.0.0.1:56356 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 07:03:07.289904 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.259724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.0.97.61:41691)
0620 07:03:07.259920 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.259927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.285844 (+ 25917us) server_negotiation.cc:407] Connection header received
0620 07:03:07.288485 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.288491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.288588 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.288653 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.289661 (+ 1008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40876: BlockingRecv error: recv got EOF from 127.0.0.1:40876 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 07:03:07.292241 23419 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.269432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.97.60:35357)
0620 07:03:07.269669 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.269675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.290648 (+ 20973us) server_negotiation.cc:407] Connection header received
0620 07:03:07.290775 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.290781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.290876 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.290989 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.292006 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41220: BlockingRecv error: recv got EOF from 127.0.0.1:41220 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 07:03:07.334115 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.320756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.97.61:41691)
0620 07:03:07.320944 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.320951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.331815 (+ 10864us) server_negotiation.cc:407] Connection header received
0620 07:03:07.332406 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.332413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.332602 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.332698 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.333802 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40888: BlockingRecv error: recv got EOF from 127.0.0.1:40888 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 07:03:07.347393 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.345069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40896 (local address 127.0.97.61:41691)
0620 07:03:07.345247 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.345258 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.345828 (+ 570us) server_negotiation.cc:407] Connection header received
0620 07:03:07.346076 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.346082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.346209 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.346321 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.347158 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40896: BlockingRecv error: recv got EOF from 127.0.0.1:40896 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 07:03:07.355618 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.353679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.0.97.61:41691)
0620 07:03:07.353858 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.353864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.353970 (+ 106us) server_negotiation.cc:407] Connection header received
0620 07:03:07.354266 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.354272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.354382 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.354493 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.355363 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40906: BlockingRecv error: recv got EOF from 127.0.0.1:40906 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 07:03:07.365566 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.363264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.0.97.61:41691)
0620 07:03:07.363488 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.363494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.364036 (+ 542us) server_negotiation.cc:407] Connection header received
0620 07:03:07.364246 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.364252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.364353 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.364480 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.365311 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40914: BlockingRecv error: recv got EOF from 127.0.0.1:40914 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260620 07:03:07.379935 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.377440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.0.97.61:41691)
0620 07:03:07.377674 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.377680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.378250 (+ 570us) server_negotiation.cc:407] Connection header received
0620 07:03:07.378531 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.378566 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.378702 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.378808 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.379679 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40920: BlockingRecv error: recv got EOF from 127.0.0.1:40920 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 07:03:07.403313 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.400681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.0.97.61:41691)
0620 07:03:07.400897 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.400904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.401463 (+ 559us) server_negotiation.cc:407] Connection header received
0620 07:03:07.401762 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.401769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.401872 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.402001 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.402918 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40926: BlockingRecv error: recv got EOF from 127.0.0.1:40926 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 07:03:07.441699 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.439017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.0.97.61:41691)
0620 07:03:07.439228 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.439235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.439768 (+ 533us) server_negotiation.cc:407] Connection header received
0620 07:03:07.439983 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.439989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.440102 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.440216 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.441416 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40938: BlockingRecv error: recv got EOF from 127.0.0.1:40938 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 07:03:07.511832 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.509620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.0.97.61:41691)
0620 07:03:07.509867 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.509874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.510325 (+ 451us) server_negotiation.cc:407] Connection header received
0620 07:03:07.510494 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.510499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.510634 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.510754 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.511586 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40940: BlockingRecv error: recv got EOF from 127.0.0.1:40940 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 07:03:07.648000 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.645881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.0.97.61:41691)
0620 07:03:07.646139 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.646145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.646473 (+ 328us) server_negotiation.cc:407] Connection header received
0620 07:03:07.646693 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.646698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.646783 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.646869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.647743 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40954: BlockingRecv error: recv got EOF from 127.0.0.1:40954 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 07:03:07.788924 23393 catalog_manager.cc:1259] Loaded cluster ID: 404f044d482541dc9d8ff423b1bb600e
I20260620 07:03:07.789026 23393 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: loading cluster ID for follower catalog manager: success
I20260620 07:03:07.791549 23393 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: acquiring CA information for follower catalog manager: success
I20260620 07:03:07.793365 23393 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 867efa4ee06b48ec875110aba103eb1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 07:03:07.914224 23306 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 07:03:07.912152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.0.97.61:41691)
0620 07:03:07.912395 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:07.912402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:07.912648 (+ 246us) server_negotiation.cc:407] Connection header received
0620 07:03:07.912849 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:07.912855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:07.912941 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:07.913033 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 07:03:07.913987 (+ 954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40958: BlockingRecv error: recv got EOF from 127.0.0.1:40958 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 07:03:08.094362 23442 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: acquiring CA information for follower catalog manager: success
I20260620 07:03:08.096477 23442 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65b7430b2c824ac98c85528148e48413: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 07:03:08.434789 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:08.429362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.0.97.61:41691)
0620 07:03:08.429825 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:08.429829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:08.429849 (+ 20us) server_negotiation.cc:407] Connection header received
0620 07:03:08.430020 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:08.430024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:08.430173 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:08.430282 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:08.430665 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.431374 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.432687 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.433098 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.433203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:08.433652 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:08.434197 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:08.434322 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:08.434406 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 07:03:08.434474 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
I20260620 07:03:08.445746 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:08.440328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.97.60:35357)
0620 07:03:08.440895 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:08.440899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:08.440915 (+ 16us) server_negotiation.cc:407] Connection header received
0620 07:03:08.441002 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:08.441006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:08.441151 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:08.441301 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:08.441620 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.442412 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.443672 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.444083 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.444181 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:08.444625 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:08.445171 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:08.445283 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:08.445346 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 07:03:08.445423 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":371,"threads_started":1}
I20260620 07:03:08.460341 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:08.452003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.0.97.60:35357)
0620 07:03:08.454047 (+ 2044us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:08.454051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:08.454065 (+ 14us) server_negotiation.cc:407] Connection header received
0620 07:03:08.454230 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:08.454235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:08.454318 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:08.454417 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:08.455125 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.456031 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.457827 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.458449 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.458601 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:08.459477 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:08.459613 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:08.459689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:08.459855 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 07:03:08.459933 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1948}
I20260620 07:03:08.460489 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:08.451645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.0.97.61:41691)
0620 07:03:08.451863 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:08.451869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:08.454727 (+ 2858us) server_negotiation.cc:407] Connection header received
0620 07:03:08.454882 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:08.454886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:08.454974 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:08.455062 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:08.455740 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.456751 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.458668 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.459106 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.459200 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:08.459753 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:08.459879 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:08.459969 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:08.460160 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 07:03:08.460239 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 07:03:08.461620 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 07:03:08.451367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.0.97.62:41045)
0620 07:03:08.454317 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0620 07:03:08.454323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 07:03:08.454340 (+ 17us) server_negotiation.cc:407] Connection header received
0620 07:03:08.454452 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 07:03:08.454456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 07:03:08.454709 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 07:03:08.454952 (+ 243us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 07:03:08.455536 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.456560 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.458338 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 07:03:08.458874 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 07:03:08.458999 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 07:03:08.459478 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 07:03:08.459897 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 07:03:08.460000 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 07:03:08.461291 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0620 07:03:08.461389 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1841,"thread_start_us":1773,"threads_started":1}
I20260620 07:03:08.465569 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 23160
I20260620 07:03:08.466499 23222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:08.575682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 23160
I20260620 07:03:08.593462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 23234
I20260620 07:03:08.594528 23296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:08.700711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 23234
I20260620 07:03:08.716921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task8UMGcc/build/asan/bin/kudu pid 23309
I20260620 07:03:08.718003 23371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 07:03:08.823369 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8UMGcc/build/asan/bin/kudu with pid 23309
2026-06-20T07:03:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5047 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9887 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (354006 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST